T5

T5 means Text-to-Text Transfer Transformer. It uses a Transformer encoder and decoder. It represents each task as text input and text output, including classification.

Text-to-text tasks

Task	Input	Output
Translation	Text and a translation instruction	Translated text
Summarisation	Text and a summary instruction	Summary text
Classification	Text and a classification instruction	Class label written as text

Denoising with 15 percent masking

1. Select 15 percent of the original input tokens for corruption
2. Group consecutive selected tokens into spans
3. Replace each span with one sentinel token
4. Feed the remaining text and sentinels into the encoder
5. Train the decoder to generate the missing spans in order

A sentinel is a special vocabulary token that marks one missing span. One sentinel can stand for several tokens. The decoder must learn both the missing content and the length of that content.

encoder input:  remaining text + sentinel markers
                 -> encoder representations
                 -> decoder generates sentinels + missing spans

T5 generates the missing spans. BART reconstructs the complete original text.

Splitting text for generation

Prefix language modelling gives the encoder the first part of a text. The decoder generates the second part one token at a time.

1. Choose a random split point in the input text
2. Send the first part, or prefix, to the encoder
3. Use the second part as the decoder's target
4. Train the decoder to generate that target autoregressively

This is an alternative training objective examined in the T5 work. T5's final denoising objective uses span corruption. T5 paper

See Autoregressive and Autoencoding Objectives for the difference between text reconstruction and next-token generation.