A pre-training objective defines what a language model learns to predict. Two main choices are autoregressive prediction and autoencoding through reconstruction.
Autoregressive objective
An autoregressive (AR) model predicts a token from tokens that precede it in the prediction order.
- Standard GPT models predict from left to right
- A causal attention mask limits access to future tokens
- A decoder generates the output one token at a time
- The same next-token task supports text generation after pre-training
Causal Language Modelling explains the shifted input and target used for this objective.
Autoencoding objective
An autoencoding (AE) language model reconstructs original content from corrupted text.
- Corruption can replace selected tokens with masks
- Bidirectional context helps the model recover the selected tokens
- BERT-style Masked Language Modelling predicts selected targets separately, given the corrupted input
- This objective supports language-understanding tasks
Pre-training and fine-tuning mismatch
| Stage | BERT-style masked objective | Standard autoregressive objective |
|---|---|---|
| Pre-training input | Text that includes artificial masks or other replacements | Earlier tokens in a sequence |
| Prediction task | Recover selected original tokens | Predict the next token |
| Later task | Often normal text for classification, tagging, or question answering | Can use next-token generation for the task answer |
For BERT, fine-tuning normally receives complete text while pre-training receives corrupted text. This creates an input mismatch.
For text generation, there is also an objective mismatch: reconstructing masked positions differs from generating a new sequence one token at a time. Autoregressive pre-training is more closely matched to generation.
A matching objective reduces this discrepancy. Task-specific fine-tuning can still be useful.
How other models use these ideas
- T5 and BART combine a denoising input with an autoregressive decoder
- XLNet uses different autoregressive prediction orders to learn from context on both sides, while avoiding inserted mask tokens
The objective and architecture are separate choices. An encoder-decoder model can use denoising training and still generate its output autoregressively.