Transformer-based language models use different parts of the Transformer and different pre-training objectives. These choices affect whether a model is strongest at text understanding, text generation, or both.

Pre-training and fine-tuning

Pre-training learns broad language patterns from a large and diverse collection of unlabelled text.

Fine-tuning continues training on a smaller labelled dataset for a specific task such as sentiment classification, question answering, or named entity recognition.

large unlabelled corpus -> pre-trained model -> task dataset -> fine-tuned model

Main architecture and objective choices

FamilyTransformer partMain objectiveMain strength
Encoder-onlyEncoder stackReconstruct masked or corrupted textBidirectional understanding
Decoder-onlyDecoder stackPredict the next token from left to rightAutoregressive generation
Encoder-decoderEncoder and decoderEncode source or corrupted text, then generate target textText-to-text tasks

Autoregressive objective

An autoregressive model predicts later text from earlier text. The decoder uses a causal mask, so each position can attend only to the current and earlier positions.

This objective matches text generation because pre-training and generation both use left-to-right prediction. This reduces the mismatch between pre-training and generation tasks.

Autoencoding objective

An autoencoding model receives corrupted text and reconstructs the original content. Masked language modelling is one example.

The encoder can use context from both sides of a masked position. This design supports language-understanding tasks.

Masked pre-training input differs from the normal text used during fine-tuning. Predicting masked positions also differs from generating a new sequence one token at a time. See Autoregressive and Autoencoding Objectives for this mismatch and the way XLNet addresses it.

Position and token representation

The original Transformer adds absolute sinusoidal positional encodings to token embeddings. Transformer-XL uses relative positions so attention can represent the distance between tokens.

Large language models commonly use subword tokenisation. Common methods include WordPiece and BPE. Subwords reduce the unknown-word problem while keeping the vocabulary manageable.

BERT family

BERT uses an encoder stack and two pre-training tasks.

Masked Language Modelling

For Masked Language Modelling, BERT selects 15 percent of its input tokens:

  • 80 percent of the selected tokens become [MASK]
  • 10 percent become random tokens
  • 10 percent remain unchanged

The 80/10/10 split applies to the selected 15 percent. The model predicts the original selected tokens from bidirectional context.

Next Sentence Prediction

In Next Sentence Prediction, BERT receives two sentences and predicts whether the second sentence follows the first. Half of the training pairs are consecutive sentences, and half use a random second sentence.

Fine-tuning uses

BERT fine-tuning supports four main task types:

  • Sentence-pair classification
  • Single-sentence classification
  • Question answering
  • Single-sentence tagging, such as named entity recognition

RoBERTa and distilled variants

RoBERTa changes the BERT training procedure:

  • Removes Next Sentence Prediction
  • Uses dynamic masking across epochs
  • Uses BPE instead of WordPiece
  • Trains with more data and longer sequences

DistilBERT trains a smaller student model to imitate the output probabilities of a larger BERT teacher. It uses half as many encoder layers. It is 40 percent smaller, runs 60 percent faster, and retains 97 percent of BERT's language-understanding capability.

DistilRoBERTa applies the same teacher-student idea with RoBERTa as the teacher.

GPT family

GPT uses a decoder-only architecture and left-to-right autoregressive pre-training.

The GPT family developed through this progression:

ModelYearParameter countMain development
GPT-12018About 117 millionRelied on downstream fine-tuning
GPT-22019Up to about 1.5 billionStronger in-context task behaviour and byte-level BPE
GPT-32020Up to 175 billionStrong zero-shot and few-shot in-context learning
GPT-42023Public count undisclosedText and image input

The GPT-4 estimate of 1.7 trillion parameters is unconfirmed. The public technical report withholds the model size and detailed training-data information. GPT-4 technical report

In-context learning and fine-tuning

MethodTask informationWeight update
Zero-shotInstruction with no exampleNo
One-shotInstruction with one exampleNo
Few-shotInstruction with several examplesNo
Fine-tuningTask datasetYes

In-context learning changes the prompt and the model activations. Fine-tuning changes model parameters.

InstructGPT and human feedback

InstructGPT first learns from human-written demonstrations. In RLHF, human rankings train a reward model. Its scores then guide reinforcement-learning updates to the language model.

ChatGPT uses a related method for dialogue. The original release was fine-tuned from GPT-3.5. The simplified description, "GPT-3 fine-tuned with InstructGPT", refers to this demonstration and human-feedback training approach. Original ChatGPT announcement

Encoder-decoder models

T5

T5 converts every task into text input and text output. Its denoising pre-training selects 15 percent of tokens and replaces consecutive selected spans with sentinel tokens. The decoder generates the missing spans.

A separate prefix language modelling objective splits text at a random point: the first part goes to the encoder, and the decoder generates the second part autoregressively. T5 examined this alternative; its final denoising objective uses span corruption. T5 paper

BART

BART uses a bidirectional encoder and an autoregressive decoder. Its denoising objective masks 30 percent of tokens and permutes sentence order. The decoder reconstructs the original text.

Transformer-XL and XLNet

Transformer-XL caches hidden states from an earlier segment and reuses them when it processes the next segment. This segment-level recurrence supports dependencies that extend beyond one fixed segment. It also uses relative positional information.

XLNet uses permutation language modelling. It learns to predict tokens under different factorisation orders while keeping autoregressive prediction. It is based on Transformer-XL and uses segment recurrence and relative positions. It addresses the left-to-right context limit and the mismatch caused by masked pre-training input.

Main comparison

ModelArchitectureKey learning method
BERTEncoder-onlyMasked language modelling and Next Sentence Prediction
RoBERTaEncoder-onlyBERT-style masking without Next Sentence Prediction
DistilBERTEncoder-only studentDistillation from BERT
DistilRoBERTaEncoder-only studentDistillation from RoBERTa
GPTDecoder-onlyLeft-to-right next-token prediction
T5Encoder-decoderSpan corruption and text-to-text generation
BARTEncoder-decoderDenoising and autoregressive reconstruction
Transformer-XLAutoregressive Transformer with memorySegment recurrence and relative positions
XLNetTransformer-XL basedPermutation language modelling

Related notes: Transformers · Attention Masks · Causal Language Modelling · In-Context Learning · LLM Training Stages · Tokenization Schemes

Code

Hugging Face: fine-tune BERT for five-class text classification

import numpy as np
import torch
import evaluate
from datasets import load_dataset
from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
    Trainer,
    TrainingArguments,
)
 
dataset = load_dataset("yelp_review_full")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
 
def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        padding="max_length",
        truncation=True,
    )
 
tokenized = dataset.map(tokenize_function, batched=True)
train_data = tokenized["train"].shuffle(seed=42).select(range(1000))
eval_data = tokenized["test"].shuffle(seed=42).select(range(1000))
 
model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-cased",
    num_labels=5,
)
 
accuracy = evaluate.load("accuracy")
 
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return accuracy.compute(predictions=predictions, references=labels)
 
training_args = TrainingArguments(
    output_dir="test_trainer",
    evaluation_strategy="epoch",
)
 
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_data,
    eval_dataset=eval_data,
    compute_metrics=compute_metrics,
)
 
trainer.train()

The tokenizer returns token IDs and an attention mask. The attention mask marks real tokens and padding positions. AutoModelForSequenceClassification adds a classification head to BERT. With num_labels=5, the model returns five logits for each review.

PyTorch: run one prediction

model.eval()
 
example = tokenizer(
    "The food was excellent.",
    return_tensors="pt",
    padding=True,
    truncation=True,
)
 
with torch.no_grad():
    logits = model(**example).logits
 
predicted_class = logits.argmax(dim=-1)
print(tuple(logits.shape))        # (1, 5)
print(predicted_class.item())

model.eval() selects evaluation behaviour for dropout and similar layers. It does not freeze parameters. torch.no_grad() prevents gradient storage during this prediction. The output still contains raw logits, and argmax selects the largest class score.

An LLM keeps its knowledge frozen in its parameters, and Retrieval Augmented Generation (RAG) adds outside knowledge at inference time.