Transformer-based language models use different parts of the Transformer and different pre-training objectives. These choices affect whether a model is strongest at text understanding, text generation, or both.
Pre-training and fine-tuning
Pre-training learns broad language patterns from a large and diverse collection of unlabelled text.
Fine-tuning continues training on a smaller labelled dataset for a specific task such as sentiment classification, question answering, or named entity recognition.
large unlabelled corpus -> pre-trained model -> task dataset -> fine-tuned modelMain architecture and objective choices
| Family | Transformer part | Main objective | Main strength |
|---|---|---|---|
| Encoder-only | Encoder stack | Reconstruct masked or corrupted text | Bidirectional understanding |
| Decoder-only | Decoder stack | Predict the next token from left to right | Autoregressive generation |
| Encoder-decoder | Encoder and decoder | Encode source or corrupted text, then generate target text | Text-to-text tasks |
Autoregressive objective
An autoregressive model predicts later text from earlier text. The decoder uses a causal mask, so each position can attend only to the current and earlier positions.
This objective matches text generation because pre-training and generation both use left-to-right prediction. This reduces the mismatch between pre-training and generation tasks.
Autoencoding objective
An autoencoding model receives corrupted text and reconstructs the original content. Masked language modelling is one example.
The encoder can use context from both sides of a masked position. This design supports language-understanding tasks.
Masked pre-training input differs from the normal text used during fine-tuning. Predicting masked positions also differs from generating a new sequence one token at a time. See Autoregressive and Autoencoding Objectives for this mismatch and the way XLNet addresses it.
Position and token representation
The original Transformer adds absolute sinusoidal positional encodings to token embeddings. Transformer-XL uses relative positions so attention can represent the distance between tokens.
Large language models commonly use subword tokenisation. Common methods include WordPiece and BPE. Subwords reduce the unknown-word problem while keeping the vocabulary manageable.
BERT family
BERT uses an encoder stack and two pre-training tasks.
Masked Language Modelling
For Masked Language Modelling, BERT selects 15 percent of its input tokens:
- 80 percent of the selected tokens become
[MASK] - 10 percent become random tokens
- 10 percent remain unchanged
The 80/10/10 split applies to the selected 15 percent. The model predicts the original selected tokens from bidirectional context.
Next Sentence Prediction
In Next Sentence Prediction, BERT receives two sentences and predicts whether the second sentence follows the first. Half of the training pairs are consecutive sentences, and half use a random second sentence.
Fine-tuning uses
BERT fine-tuning supports four main task types:
- Sentence-pair classification
- Single-sentence classification
- Question answering
- Single-sentence tagging, such as named entity recognition
RoBERTa and distilled variants
RoBERTa changes the BERT training procedure:
- Removes Next Sentence Prediction
- Uses dynamic masking across epochs
- Uses BPE instead of WordPiece
- Trains with more data and longer sequences
DistilBERT trains a smaller student model to imitate the output probabilities of a larger BERT teacher. It uses half as many encoder layers. It is 40 percent smaller, runs 60 percent faster, and retains 97 percent of BERT's language-understanding capability.
DistilRoBERTa applies the same teacher-student idea with RoBERTa as the teacher.
GPT family
GPT uses a decoder-only architecture and left-to-right autoregressive pre-training.
The GPT family developed through this progression:
| Model | Year | Parameter count | Main development |
|---|---|---|---|
| GPT-1 | 2018 | About 117 million | Relied on downstream fine-tuning |
| GPT-2 | 2019 | Up to about 1.5 billion | Stronger in-context task behaviour and byte-level BPE |
| GPT-3 | 2020 | Up to 175 billion | Strong zero-shot and few-shot in-context learning |
| GPT-4 | 2023 | Public count undisclosed | Text and image input |
The GPT-4 estimate of 1.7 trillion parameters is unconfirmed. The public technical report withholds the model size and detailed training-data information. GPT-4 technical report
In-context learning and fine-tuning
| Method | Task information | Weight update |
|---|---|---|
| Zero-shot | Instruction with no example | No |
| One-shot | Instruction with one example | No |
| Few-shot | Instruction with several examples | No |
| Fine-tuning | Task dataset | Yes |
In-context learning changes the prompt and the model activations. Fine-tuning changes model parameters.
InstructGPT and human feedback
InstructGPT first learns from human-written demonstrations. In RLHF, human rankings train a reward model. Its scores then guide reinforcement-learning updates to the language model.
ChatGPT uses a related method for dialogue. The original release was fine-tuned from GPT-3.5. The simplified description, "GPT-3 fine-tuned with InstructGPT", refers to this demonstration and human-feedback training approach. Original ChatGPT announcement
Encoder-decoder models
T5
T5 converts every task into text input and text output. Its denoising pre-training selects 15 percent of tokens and replaces consecutive selected spans with sentinel tokens. The decoder generates the missing spans.
A separate prefix language modelling objective splits text at a random point: the first part goes to the encoder, and the decoder generates the second part autoregressively. T5 examined this alternative; its final denoising objective uses span corruption. T5 paper
BART
BART uses a bidirectional encoder and an autoregressive decoder. Its denoising objective masks 30 percent of tokens and permutes sentence order. The decoder reconstructs the original text.
Transformer-XL and XLNet
Transformer-XL caches hidden states from an earlier segment and reuses them when it processes the next segment. This segment-level recurrence supports dependencies that extend beyond one fixed segment. It also uses relative positional information.
XLNet uses permutation language modelling. It learns to predict tokens under different factorisation orders while keeping autoregressive prediction. It is based on Transformer-XL and uses segment recurrence and relative positions. It addresses the left-to-right context limit and the mismatch caused by masked pre-training input.
Main comparison
| Model | Architecture | Key learning method |
|---|---|---|
| BERT | Encoder-only | Masked language modelling and Next Sentence Prediction |
| RoBERTa | Encoder-only | BERT-style masking without Next Sentence Prediction |
| DistilBERT | Encoder-only student | Distillation from BERT |
| DistilRoBERTa | Encoder-only student | Distillation from RoBERTa |
| GPT | Decoder-only | Left-to-right next-token prediction |
| T5 | Encoder-decoder | Span corruption and text-to-text generation |
| BART | Encoder-decoder | Denoising and autoregressive reconstruction |
| Transformer-XL | Autoregressive Transformer with memory | Segment recurrence and relative positions |
| XLNet | Transformer-XL based | Permutation language modelling |
Related notes: Transformers · Attention Masks · Causal Language Modelling · In-Context Learning · LLM Training Stages · Tokenization Schemes
Code
Hugging Face: fine-tune BERT for five-class text classification
import numpy as np
import torch
import evaluate
from datasets import load_dataset
from transformers import (
AutoModelForSequenceClassification,
AutoTokenizer,
Trainer,
TrainingArguments,
)
dataset = load_dataset("yelp_review_full")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
def tokenize_function(examples):
return tokenizer(
examples["text"],
padding="max_length",
truncation=True,
)
tokenized = dataset.map(tokenize_function, batched=True)
train_data = tokenized["train"].shuffle(seed=42).select(range(1000))
eval_data = tokenized["test"].shuffle(seed=42).select(range(1000))
model = AutoModelForSequenceClassification.from_pretrained(
"bert-base-cased",
num_labels=5,
)
accuracy = evaluate.load("accuracy")
def compute_metrics(eval_pred):
logits, labels = eval_pred
predictions = np.argmax(logits, axis=-1)
return accuracy.compute(predictions=predictions, references=labels)
training_args = TrainingArguments(
output_dir="test_trainer",
evaluation_strategy="epoch",
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_data,
eval_dataset=eval_data,
compute_metrics=compute_metrics,
)
trainer.train()The tokenizer returns token IDs and an attention mask. The attention mask marks real tokens and padding positions. AutoModelForSequenceClassification adds a classification head to BERT. With num_labels=5, the model returns five logits for each review.
PyTorch: run one prediction
model.eval()
example = tokenizer(
"The food was excellent.",
return_tensors="pt",
padding=True,
truncation=True,
)
with torch.no_grad():
logits = model(**example).logits
predicted_class = logits.argmax(dim=-1)
print(tuple(logits.shape)) # (1, 5)
print(predicted_class.item())model.eval() selects evaluation behaviour for dropout and similar layers. It does not freeze parameters. torch.no_grad() prevents gradient storage during this prediction. The output still contains raw logits, and argmax selects the largest class score.
An LLM keeps its knowledge frozen in its parameters, and Retrieval Augmented Generation (RAG) adds outside knowledge at inference time.