XLNet is a language model introduced by Carnegie Mellon University and Google in 2019. It builds on Transformer-XL and trains with an objective called permutation language modelling.

The goal of XLNet is to get context from both sides of a token, like BERT, but keep the autoregressive (AR) prediction of GPT. It does this without [MASK] tokens.

XLNet is a decoder-only model. It has no separate encoder and decoder.

Why XLNet

Earlier objectiveLimitationXLNet approach
Left-to-right AR prediction (GPT)Each prediction uses only the left contextPredicts tokens in many different orders, so each token can see both sides
Denoising with masks (BERT)Pre-training input has [MASK] tokens, but fine-tuning input does notKeeps AR prediction, so no [MASK] tokens are inserted

See Autoregressive and Autoencoding Objectives for the pre-training and fine-tuning mismatch.

Permutation language modelling

A normal AR model predicts tokens in the order 1, 2, 3, 4, 5. XLNet picks a random factorisation order, which is the order in which the positions are predicted. Each token is predicted from the tokens that come earlier in this order, not earlier in the sentence.

Example with the sentence "New York is a city" and the order 3, 2, 5, 1, 4:

StepPosition predictedToken predictedTokens it can see
13isnone
22Yorkis
35cityis, York
41Newis, York, city
54ais, York, city, New

At step 4, the model predicts "New" from "York", "is" and "city". These tokens are all on the right of "New". A left-to-right model can never do this.

What happens across many orders:

  • Training samples a new order for each sequence
  • Over many orders, each token is predicted from different sets of left and right tokens
  • The model therefore learns context from both sides, but each single prediction is still AR

The permutation changes only the prediction order. The input sentence is not shuffled, and each token keeps its original position. An attention mask controls which tokens each prediction can see.

Same seats, new speaking order

Think of the tokens as people in fixed seats. XLNet does not move anyone to a new seat. It only changes who speaks first, so each person hears a different group before they speak.

Training objective

What each part means:

  • is one factorisation order, picked from , the set of all orders of a sequence with length
  • is the token at step of that order
  • are the tokens that come before step in that order
  • means XLNet maximises the average log-likelihood over all orders

If is always , this becomes the normal left-to-right AR objective.

Transformer-XL backbone

XLNet uses two parts of Transformer-XL:

  • Segment recurrence: reuse hidden states from earlier text segments, so XLNet can handle long text
  • Relative positional encoding: represent distances between token positions, which works across segments

Comparison with BERT and GPT

ModelObjectiveContext for a predictionUses [MASK] tokens
GPTLeft-to-right ARLeft side onlyNo
BERTMasked denoising (AE)Both sidesYes
XLNetPermutation ARBoth sides, across many ordersNo

Code

These are the steps to use XLNet for a multiple choice task with Hugging Face.

Load the tokenizer and the model:

from transformers import AutoTokenizer, XLNetForMultipleChoice
 
tokenizer = AutoTokenizer.from_pretrained("xlnet/xlnet-base-cased")
model = XLNetForMultipleChoice.from_pretrained("xlnet/xlnet-base-cased").to(device).eval()

The warning about logits_proj and sequence_summary weights tells us that the multiple choice head is new. It has random weights, so we must fine-tune it before we can trust the answers.

Pair the question with each answer and pick the best one:

question = "What does NLP stands for"
answers = ["Computer Vision", "Natural Language Processing"]
 
encoding = tokenizer([question] * len(answers), answers, return_tensors="pt", padding=True).to(device)
with torch.no_grad():
    outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()})
    logits = outputs.logits
    softmax = torch.nn.functional.softmax(logits, dim=-1)
    index = torch.argmax(softmax, dim=-1)
    print("Answer:", answers[index])

How the code works:

  • [question] * len(answers) repeats the question once for each answer, so the tokenizer makes one question and answer pair for each choice
  • The model expects the shape (batch, number of choices, sequence length), so unsqueeze(0) adds the batch dimension
  • The model gives 1 score (logit) for each choice, and argmax picks the choice with the highest score