BART

BART is a Transformer model with a bidirectional encoder and an autoregressive decoder. Facebook introduced it in 2019.

Its denoising objective trains it to reconstruct original text from corrupted text.

Pre-training process

1. Mask 30 percent of the tokens in the training text
2. Permute the sentence order, which means change the order of sentences
3. Feed the corrupted text into the bidirectional encoder
4. Use the decoder to reconstruct the complete original text, one token at a time

original text -> mask tokens and permute sentences
              -> encoder reads corrupted text
              -> decoder reconstructs original text

The encoder provides context from the corrupted input. The decoder produces the reconstructed text autoregressively.

Comparison with T5

Property	T5	BART
Architecture	Encoder-decoder	Encoder-decoder
Corrupted token proportion	15 percent	30 percent
Main corruption	Replace missing spans with sentinels	Mask text and permute sentences
Denoising target	Missing spans	Complete original text

Both models combine input understanding with text generation. See Autoregressive and Autoencoding Objectives.