Hi, I followed the steps mentioned to install, but meet the following error when trying to run the pre-training command. ImportError: cannot import name 'AutoTokenizer' from 'transformers' (unknown location) P.S: I was able to install this library and do the pre-training a few weeks back but I tried to do it again and see the above error.

Thanks for your work.

Hi @prajdabre ,

Thanks for doing great work with this library. I was trying to use it and ran into this issue where prompt_params, adaptor_layers, deep_adaptor_tuning, deep_adaptor_tuning_ffn_only, parallel_adaptors params are being passed here to forward here but the MBartForConditionalGeneration class's forward function doesn't expect it.

Wanted to understand from you if the fix is as simple as creating these params in forward function call with default value of None (in which case I'm guessing we would need to make changes in the forward functions implementation itself to use these params).

Let me know if you think I might be missing something here. Thanks!

The command we are using: python pretrain_nmt.py -n 1 -nr 0 -g 1 --use_official_pretrained --pretrained_model ai4bharat/IndicBART --tokenizer_name_or_path ai4bharat/IndicBART --langs kn --mono_src /home/aniruddha/all_data/train.kn --batch_size 8 --batch_size_indicates_lines --shard_files --model_path aibharat/IndicBART/model --port 7878

Traceback (most recent call last): File "pretrain_nmt.py", line 970, in run_demo() File "pretrain_nmt.py", line 967, in run_demo mp.spawn(model_create_load_run_save, nprocs=args.gpus, args=(args,files,train_files,)) # File "/home/aniruddha/anaconda3/envs/pretam/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/home/aniruddha/anaconda3/envs/pretam/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes while not context.join(): File "/home/aniruddha/anaconda3/envs/pretam/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join raise Exception(msg) Exception:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/aniruddha/anaconda3/envs/pretam/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in wrap fn(i, *args) File "/home/aniruddha/yanmtt/pretrain_nmt.py", line 521, in model_create_load_run_save lprobs, labels, args.label_smoothing, ignore_index=tok.pad_token_id File "/home/aniruddha/yanmtt/common_utils.py", line 147, in label_smoothed_nll_loss smooth_loss.masked_fill(pad_mask, 0.0) RuntimeError: The expanded size of the tensor (333) must match the existing size (332) at non-singleton dimension 1. Target sizes: [8, 333, 1]. Tensor sizes: [8, 332, 1]

But when the data file has ".hi " language extension the code works fine.

After I follow the installation and run examples/train_mbart_model.sh, I get the below error.

Loading from checkpoint Traceback (most recent call last): File "pretrain_nmt.py", line 630, in run_demo() File "pretrain_nmt.py", line 627, in run_demo mp.spawn(model_create_load_run_save, nprocs=args.gpus, args=(args,files,train_files,)) # File "/root/yanmtt/py36/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/root/yanmtt/py36/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes while not context.join(): File "/root/yanmtt/py36/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join raise Exception(msg) Exception:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/root/yanmtt/py36/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, *args) File "/root/yanmtt/pretrain_nmt.py", line 359, in model_create_load_run_save if mod_compute.additional_lm_logits is not None: AttributeError: 'Seq2SeqLMOutput' object has no attribute 'additional_lm_logits'

What may be going wrong? The version of transformers I have is 4.3.2.

I am getting the following error while training on the monolingual (Hindi) corpus. I successfully trained the tokenizer on the same corpus using create_autotokenizer.sh.

Error Logs: Shuffling corpus! Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Keyword arguments {'sample': False, 'nbest': 64, 'alpha_or_dropout': 0.1} not recognized. Saving the model Loading from checkpoint Traceback (most recent call last): File "pretrain_nmt.py", line 989, in run_demo() File "pretrain_nmt.py", line 986, in run_demo mp.spawn(model_create_load_run_save, nprocs=args.gpus, args=(args,files,train_files,)) # File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes while not context.join(): File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 160, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap fn(i, *args) File "/workspace/data/yanmtt/pretrain_nmt.py", line 530, in model_create_load_run_save mod_compute = model(input_ids=input_ids, attention_mask=input_masks, decoder_input_ids=decoder_input_ids, output_hidden_states=args.distillation, output_attentions=args.distillation, label_mask=label_mask if args.num_domains_for_domain_classifier > 1 else None) ## Run the model and get logits. File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, **kwargs) File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1040, in forward output = self._run_ddp_forward(*inputs, **kwargs) File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1000, in _run_ddp_forward return module_to_run(*inputs[0], **kwargs[0]) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, **kwargs) TypeError: forward() got an unexpected keyword argument 'label_mask'

Steps to reproduce the error

conda create --name indicbart python=3.6
conda activate indicbart
pip install -r requirements.txt

The error

The conflict is caused by: 
The user requested scipy==1.5.4 
imagehash 4.2.1 depends on scipy 
missingno 0.5.0 depends on scipy 
pandas-profiling 3.1.0 depends on scipy>=1.4.1 
phik 0.12.0 depends on scipy>=1.5.2 
seaborn 0.11.1 depends on scipy>=1.0 
tensor2tensor 1.14.0 depends on scipy 
tensorflow-gpu 2.3.0 depends on scipy==1.4.1

Can you please guide as to how to resolve the dependency issues?

Hi, I'm trying to pre-train mBART model using the following parameters: !python pretrain_nmt.py -n 1 -nr 0 -g 1 --use_official_pretrained --fp16 --pretrained_model "facebook/mbart-large-50" --model_path "facebook/mbart-large-50" --tokenizer_name_or_path "facebook/mbart-large-50" --mono_src "/content/yanmtt/cleaned_Sanskrit_text_for_LM.txt" --shard_files --batch_size 16

I'm getting this error.

`Using label smoothing of 0.1 Using gradient clipping norm of 1.0 Using softmax temperature of 1.0 Masking ratio: 0.3 Training for: [''] Shuffling corpus! Zero size batch due to an abnormal example. Skipping empty batch. Zero size batch due to an abnormal example. Skipping empty batch. Zero size batch due to an abnormal example. Skipping empty batch. Saving the model Loading from checkpoint Traceback (most recent call last): File "pretrain_nmt.py", line 888, in run_demo() File "pretrain_nmt.py", line 885, in run_demo mp.spawn(model_create_load_run_save, nprocs=args.gpus, args=(args,files,train_files,)) # File "/usr/local/lib/python3.7/dist-packages/torch/multiprocessing/spawn.py", line 199, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/usr/local/lib/python3.7/dist-packages/torch/multiprocessing/spawn.py", line 157, in start_processes while not context.join(): File "/usr/local/lib/python3.7/dist-packages/torch/multiprocessing/spawn.py", line 118, in join raise Exception(msg) Exception:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, *args) File "/content/yanmtt/pretrain_nmt.py", line 488, in model_create_load_run_save lprobs, labels, args.label_smoothing, ignore_index=tok.pad_token_id File "/content/yanmtt/common_utils.py", line 130, in label_smoothed_nll_loss nll_loss = -lprobs.gather(dim=-1, index=target) RuntimeError: Size does not match at dimension 1 expected index [1, 13, 1] to be smaller than src [1, 12, 250054] apart from dimension 2`

I have aligned datasets for three different custom languages. Each corpus is a flat text file where each line is a sentence, and documents are separated by empty lines. All sentences and documents match between the datasets. There are two tasks I'd like to be able to perform: 1) translate between the languages, and 2) infill sentences from any single language. For the translation task, given languages A, B, and C, it's actually not likely I'll ever go from C -> A or B -> A, but I definitely want to translate A -> B and A -> C. Other translations that would be helpful would be B -> C and C -> B.

From the MBART examples at HuggingFace it looks like MBartForConditionalGeneration could perhaps do task 1 (though maybe not in all directions listed above?), and BartForConditionalGeneration could do task 2. But is there any reason why MBartForConditionalGeneration couldn't do both? That is, if I pass an input with a <mask> token to MBART, will it perform the infilling, just as BART would? If so, then does your toolkit make sense as a place to start?

Any thoughts very much appreciated.

I am trying to pretrain BART further from the huggingface checkpoint with the below command, and it seems like there is an issue with mismatched amount of arguments for _tokenize.

The command is below: python pretrain_nmt.py -n 1 -nr 0 -g 1 --use_official_pretrained --pretrained_model facebook/bart-large --tokenizer_name_or_path facebook/bart-large --langs en --mono_src examples/data/train.en --batch_size 8

The error is: Using softmax temperature of 1.0 Masking ratio: 0.3 Training for: ['en'] Shuffling corpus! Traceback (most recent call last): File "pretrain_nmt.py", line 628, in run_demo() File "pretrain_nmt.py", line 625, in run_demo mp.spawn(model_create_load_run_save, nprocs=args.gpus, args=(args,files,train_files,)) # File "/root/yanmtt/py36/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/root/yanmtt/py36/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes while not context.join(): File "/root/yanmtt/py36/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join raise Exception(msg) Exception:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/root/yanmtt/py36/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap fn(i, *args) File "/root/yanmtt/pretrain_nmt.py", line 221, in model_create_load_run_save for input_ids, input_masks, decoder_input_ids, labels in generate_batches_monolingual_masked_or_bilingual(tok, args, rank, files, train_files, ctr): #Batches are generated from here. The argument (0.30, 0.40) is a range which indicates the percentage of the source sentence to be masked in case we want masking during training just like we did during BART pretraining. The argument 3.5 is the lambda to the poisson length sampler which indicates the average length of a word sequence that will be masked. Since this is pretraining we do not do any evaluations even if we train on parallel corpora. File "/root/yanmtt/common_utils.py", line 482, in generate_batches_monolingual_masked iids = tok(lang + " " + masked_sentence + " ", add_special_tokens=False, return_tensors="pt").input_ids File "/root/yanmtt/py36/lib/python3.6/site-packages/transformers-4.3.2-py3.6.egg/transformers/tokenization_utils_base.py", line 2377, in call **kwargs, File "/root/yanmtt/py36/lib/python3.6/site-packages/transformers-4.3.2-py3.6.egg/transformers/tokenization_utils_base.py", line 2447, in encode_plus **kwargs, File "/root/yanmtt/py36/lib/python3.6/site-packages/transformers-4.3.2-py3.6.egg/transformers/tokenization_utils.py", line 441, in _encode_plus first_ids = get_input_ids(text) File "/root/yanmtt/py36/lib/python3.6/site-packages/transformers-4.3.2-py3.6.egg/transformers/tokenization_utils.py", line 410, in get_input_ids tokens = self.tokenize(text, **kwargs) File "/root/yanmtt/py36/lib/python3.6/site-packages/transformers-4.3.2-py3.6.egg/transformers/tokenization_utils.py", line 342, in tokenize tokenized_text = split_on_tokens(no_split_token, text) File "/root/yanmtt/py36/lib/python3.6/site-packages/transformers-4.3.2-py3.6.egg/transformers/tokenization_utils.py", line 336, in split_on_tokens for token in tokenized_text File "/root/yanmtt/py36/lib/python3.6/site-packages/transformers-4.3.2-py3.6.egg/transformers/tokenization_utils.py", line 336, in for token in tokenized_text TypeError: _tokenize() takes 2 positional arguments but 5 were given

Upon some further inspection, it seems like in a commit a few days ago, this line was changed to have 4 arguments: https://github.com/prajdabre/yanmtt/blob/main/transformers/src/transformers/tokenization_utils.py#L319

However, the _tokenize function for BART tokenizer (which inherits all the way down from GPT2 I believe), takes in less arguments: https://github.com/prajdabre/yanmtt/blob/main/transformers/src/transformers/models/gpt2/tokenization_gpt2.py#L241

I run bash examples/create_tokenizer.sh and then bash examples/create_tokenizer.sh, but the latter shows

IP address is localhost
Monolingual training files are: {'hi': 'examples/data/train.hi', 'en': 'examples/data/train.en', 'vi': 'examples/data/train.vi'}
Sharding files into 1 parts
For language: hi  the total number of lines are: 18088 and number of lines per shard are: 18088
File for language hi has been sharded.
For language: en  the total number of lines are: 18088 and number of lines per shard are: 18088
File for language en has been sharded.
For language: vi  the total number of lines are: 18088 and number of lines per shard are: 18088
File for language vi has been sharded.
Sharding files into 1 parts

and then hangs without showing anything else. If I press ^C to cancel, the following traceback is shown:

  File "pretrain_nmt.py", line 888, in <module>
    run_demo()
  File "pretrain_nmt.py", line 885, in run_demo
    mp.spawn(model_create_load_run_save, nprocs=args.gpus, args=(args,files,train_files,))         #
  File "/home/user/.conda/envs/yanmtt/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/user/.conda/envs/yanmtt/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/home/user/.conda/envs/yanmtt/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 101, in join
    timeout=timeout,
  File "/home/user/.conda/envs/yanmtt/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/home/user/.conda/envs/yanmtt/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)

I am running YANMTT in a Docker container on a machine with a GPU A100 40GB. The only dependency for which I am using a newer version is torch, as the version in requirements.txt is too old for my GPU.

I am considering using YANMTT to train my own BART model. However, instead of using it as the initial model for a subsequent fine-tuning process, I am interested in using the BART model itsel to generate alternative versions of the input sentence. To do this, I would like to mask a percentage of the words in a sentence at inference time and let the model generate a variation of it via beam search decoding:

• Original sentence: Mike goes to the bookstore on Thursday
• Possible masked input sentence: <mask> goes to the bookstore <mask>
• Possible model output: Jerry happily goes to the bookstore with his friends

Can this be easily done with YANMTT? I am trying to have my own model for the generation of synthetic samples discussed in the paper "Detecting Hallucinated Content in Conditional Neural Sequence Generation" (section 3.1).

Hello!

Just wanted to know if there is an alternative to installing sentencepiece. I see it requires sudo access adding to me getting the following error:

(bart_pretraining) make install [ 10%] Built target sentencepiece_train-static Consolidate compiler generated dependencies of target sentencepiece-static [ 46%] Built target sentencepiece-static Consolidate compiler generated dependencies of target sentencepiece [ 82%] Built target sentencepiece Consolidate compiler generated dependencies of target spm_decode [ 84%] Built target spm_decode Consolidate compiler generated dependencies of target sentencepiece_train [ 93%] Built target sentencepiece_train Consolidate compiler generated dependencies of target spm_normalize [ 95%] Built target spm_normalize Consolidate compiler generated dependencies of target spm_train [ 97%] Built target spm_train Consolidate compiler generated dependencies of target spm_export_vocab [ 99%] Built target spm_export_vocab Consolidate compiler generated dependencies of target spm_encode [100%] Built target spm_encode Install the project... -- Install configuration: "" CMake Error at cmake_install.cmake:46 (file): file cannot create directory: /usr/local/lib64/pkgconfig. Maybe need administrative privileges.

when I run make -j $(nproc)

Hello,

Great repo! It's of great help to me. I just had 2 questions:

1. How do you do evaluation for pre-training?
2. Does the pre-training involve both mask infill and sentence permutation? If it does both can I just do mask infill? My main motive is to fine-tune a pre-trained BART with mask infill on an English corpus.

Thank You so much!

Currently, I have implemented the mBART (span denoising) and mT5 (span prediction) pre-training approaches but according to the ULL2 paper (https://arxiv.org/pdf/2205.05131.pdf) a more comprehensive mixture of denoisers would help a lot.

Currently, you may use either mT5 or mBART style but I would like to enable the user to specify a comma separated list of denoising objectives and a comma separated list of the probabilities of using these objectives along with requisite hyperparams for each objective. If this is done we can play with some cool stuff.

Currently the mbart backbone code I use has pre-norm which is layer(norm(input))+input whereas some people seem to say that postnorm which is norm(layer(input)+input) might be better for zeor shot. Lord alone knows whats going to be useful when.

Having a flag to control pre- and post-norm in the encoder and decoder would be perfect.