XLNet is a language model introduced by Carnegie Mellon University and Google in 2019. It builds on Transformer-XL and trains with an objective called permutation language modelling.
The goal of XLNet is to get context from both sides of a token, like BERT, but keep the autoregressive (AR) prediction of GPT. It does this without [MASK] tokens.
XLNet is a decoder-only model. It has no separate encoder and decoder.
Why XLNet
| Earlier objective | Limitation | XLNet approach |
|---|---|---|
| Left-to-right AR prediction (GPT) | Each prediction uses only the left context | Predicts tokens in many different orders, so each token can see both sides |
| Denoising with masks (BERT) | Pre-training input has [MASK] tokens, but fine-tuning input does not | Keeps AR prediction, so no [MASK] tokens are inserted |
See Autoregressive and Autoencoding Objectives for the pre-training and fine-tuning mismatch.
Permutation language modelling
A normal AR model predicts tokens in the order 1, 2, 3, 4, 5. XLNet picks a random factorisation order, which is the order in which the positions are predicted. Each token is predicted from the tokens that come earlier in this order, not earlier in the sentence.
Example with the sentence "New York is a city" and the order 3, 2, 5, 1, 4:
| Step | Position predicted | Token predicted | Tokens it can see |
|---|---|---|---|
| 1 | 3 | is | none |
| 2 | 2 | York | is |
| 3 | 5 | city | is, York |
| 4 | 1 | New | is, York, city |
| 5 | 4 | a | is, York, city, New |
At step 4, the model predicts "New" from "York", "is" and "city". These tokens are all on the right of "New". A left-to-right model can never do this.
What happens across many orders:
- Training samples a new order for each sequence
- Over many orders, each token is predicted from different sets of left and right tokens
- The model therefore learns context from both sides, but each single prediction is still AR
The permutation changes only the prediction order. The input sentence is not shuffled, and each token keeps its original position. An attention mask controls which tokens each prediction can see.
Same seats, new speaking order
Think of the tokens as people in fixed seats. XLNet does not move anyone to a new seat. It only changes who speaks first, so each person hears a different group before they speak.
Training objective
What each part means:
- is one factorisation order, picked from , the set of all orders of a sequence with length
- is the token at step of that order
- are the tokens that come before step in that order
- means XLNet maximises the average log-likelihood over all orders
If is always , this becomes the normal left-to-right AR objective.
Transformer-XL backbone
XLNet uses two parts of Transformer-XL:
- Segment recurrence: reuse hidden states from earlier text segments, so XLNet can handle long text
- Relative positional encoding: represent distances between token positions, which works across segments
Comparison with BERT and GPT
| Model | Objective | Context for a prediction | Uses [MASK] tokens |
|---|---|---|---|
| GPT | Left-to-right AR | Left side only | No |
| BERT | Masked denoising (AE) | Both sides | Yes |
| XLNet | Permutation AR | Both sides, across many orders | No |
Code
These are the steps to use XLNet for a multiple choice task with Hugging Face.
Load the tokenizer and the model:
from transformers import AutoTokenizer, XLNetForMultipleChoice
tokenizer = AutoTokenizer.from_pretrained("xlnet/xlnet-base-cased")
model = XLNetForMultipleChoice.from_pretrained("xlnet/xlnet-base-cased").to(device).eval()The warning about logits_proj and sequence_summary weights tells us that the multiple choice head is new. It has random weights, so we must fine-tune it before we can trust the answers.
Pair the question with each answer and pick the best one:
question = "What does NLP stands for"
answers = ["Computer Vision", "Natural Language Processing"]
encoding = tokenizer([question] * len(answers), answers, return_tensors="pt", padding=True).to(device)
with torch.no_grad():
outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()})
logits = outputs.logits
softmax = torch.nn.functional.softmax(logits, dim=-1)
index = torch.argmax(softmax, dim=-1)
print("Answer:", answers[index])How the code works:
[question] * len(answers)repeats the question once for each answer, so the tokenizer makes one question and answer pair for each choice- The model expects the shape (batch, number of choices, sequence length), so
unsqueeze(0)adds the batch dimension - The model gives 1 score (logit) for each choice, and
argmaxpicks the choice with the highest score