XLNet is a language model introduced by Carnegie Mellon University and Google in 2019. It builds on Transformer-XL and uses permutation language modelling.
Permutation language modelling
A standard left-to-right autoregressive objective predicts each token from earlier tokens. XLNet varies the order in which token positions are predicted.
- Train with different factorisation orders of the same text
- Predict each target using positions that come earlier in the selected prediction order
- Across these orders, learn from context on both sides of a position in the original text
The permutation changes the prediction order. Tokens keep their positions in the original text.
Problems it addresses
| Earlier objective | Limitation | XLNet approach |
|---|---|---|
| Left-to-right autoregressive prediction | Uses only left context for each prediction | Learns across different prediction orders |
| BERT-style denoising | Masked pre-training input differs from normal fine-tuning input | Uses autoregressive prediction without inserting [MASK] tokens |
See Autoregressive and Autoencoding Objectives for the pre-training and fine-tuning mismatch.
Long context
XLNet uses two parts of Transformer-XL:
- Segment recurrence: reuse hidden states from earlier text segments
- Relative positional encoding: represent distances between token positions
The training objective combines autoregressive prediction with access to bidirectional context across permutations.