Transformer-Based Large Language Models

Transformer-based language models use different parts of the Transformer and different pre-training objectives. These choices affect whether a model is strongest at text understanding, text generation, or both.

Pre-training and fine-tuning

Pre-training learns broad language patterns from a large and diverse collection of unlabelled text.

Fine-tuning continues training on a smaller labelled dataset for a specific task such as sentiment classification, question answering, or named entity recognition.

large unlabelled corpus -> pre-trained model -> task dataset -> fine-tuned model

Main architecture and objective choices

Family	Transformer part	Main objective	Main strength
Encoder-only	Encoder stack	Reconstruct masked or corrupted text	Bidirectional understanding
Decoder-only	Decoder stack	Predict the next token from left to right	Autoregressive generation
Encoder-decoder	Encoder and decoder	Encode source or corrupted text, then generate target text	Text-to-text tasks

Autoregressive objective

An autoregressive model predicts later text from earlier text. The decoder uses a causal mask, so each position can attend only to the current and earlier positions.

This objective matches text generation because pre-training and generation both use left-to-right prediction.

Autoencoding objective

An autoencoding model receives corrupted text and reconstructs the original content. Masked language modelling is one example.

The encoder can use context from both sides of a masked position. This design supports language-understanding tasks.

Position and token representation

The original Transformer adds absolute sinusoidal positional encodings to token embeddings. Transformer-XL uses relative positions so attention can represent the distance between tokens.

Large language models commonly use subword tokenisation. Common methods include WordPiece and Byte Pair Encoding, or BPE. Subwords reduce the unknown-word problem while keeping the vocabulary manageable.

BERT family

BERT uses an encoder stack and two pre-training tasks.

Masked Language Modelling

BERT selects 15 percent of its input tokens:

80 percent of the selected tokens become [MASK]
10 percent become random tokens
10 percent remain unchanged

The model predicts the original selected tokens from bidirectional context.

Next Sentence Prediction

BERT receives two sentences and predicts whether the second sentence follows the first. Half of the training pairs are consecutive sentences, and half use a random second sentence.

Fine-tuning uses

BERT fine-tuning supports four main task types:

Sentence-pair classification
Single-sentence classification
Question answering
Single-sentence tagging, such as named entity recognition

RoBERTa and distilled variants

RoBERTa changes the BERT training procedure:

Removes Next Sentence Prediction
Uses dynamic masking across epochs
Uses BPE instead of WordPiece
Trains with more data and longer sequences

DistilBERT trains a smaller student model to imitate a larger BERT teacher. It is 40 percent smaller, runs 60 percent faster, and retains 97 percent of BERT's language-understanding capability.

DistilRoBERTa applies the same teacher-student idea with RoBERTa as the teacher.

GPT family

GPT uses a decoder-only architecture and left-to-right autoregressive pre-training.

The GPT family developed through this progression:

GPT-1 relied on downstream fine-tuning
GPT-2 showed stronger in-context task behaviour
GPT-3 demonstrated zero-shot and few-shot in-context learning at a much larger scale
GPT-4 added multimodal input and can process text and images

In-context learning and fine-tuning

Method	Task information	Weight update
Zero-shot	Instruction with no example	No
One-shot	Instruction with one example	No
Few-shot	Instruction with several examples	No
Fine-tuning	Task dataset	Yes

In-context learning changes the prompt and the model activations. Fine-tuning changes model parameters.

InstructGPT and human feedback

InstructGPT first learns from human-written demonstrations. It then uses reinforcement learning from human feedback, where human rankings guide the model towards preferred outputs.

Encoder-decoder models

T5

T5 converts every task into text input and text output. During pre-training, it replaces masked spans with sentinel tokens and learns to generate the missing spans.

BART

BART uses a bidirectional encoder and an autoregressive decoder. Its denoising objective masks 30 percent of tokens and permutes sentence order. The decoder reconstructs the original text.

Transformer-XL and XLNet

Transformer-XL caches hidden states from an earlier segment and reuses them when it processes the next segment. This segment-level recurrence supports dependencies that extend beyond one fixed segment. It also uses relative positional information.

XLNet uses permutation language modelling. It learns to predict tokens under different factorisation orders while keeping autoregressive prediction. It is based on Transformer-XL and uses segment recurrence and relative positions.

Main comparison

Model	Architecture	Key learning method
BERT	Encoder-only	Masked language modelling and Next Sentence Prediction
RoBERTa	Encoder-only	BERT-style masking without Next Sentence Prediction
DistilBERT	Encoder-only student	Distillation from BERT
GPT	Decoder-only	Left-to-right next-token prediction
T5	Encoder-decoder	Span corruption and text-to-text generation
BART	Encoder-decoder	Denoising and autoregressive reconstruction
XLNet	Transformer-XL based	Permutation language modelling

Code

Hugging Face: fine-tune BERT for five-class text classification

import numpy as np
import torch
import evaluate
from datasets import load_dataset
from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
    Trainer,
    TrainingArguments,
)
 
dataset = load_dataset("yelp_review_full")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
 
def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        padding="max_length",
        truncation=True,
    )
 
tokenized = dataset.map(tokenize_function, batched=True)
train_data = tokenized["train"].shuffle(seed=42).select(range(1000))
eval_data = tokenized["test"].shuffle(seed=42).select(range(1000))
 
model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-cased",
    num_labels=5,
)
 
accuracy = evaluate.load("accuracy")
 
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return accuracy.compute(predictions=predictions, references=labels)
 
training_args = TrainingArguments(
    output_dir="test_trainer",
    evaluation_strategy="epoch",
)
 
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_data,
    eval_dataset=eval_data,
    compute_metrics=compute_metrics,
)
 
trainer.train()

The tokenizer returns token IDs and an attention mask. The attention mask marks real tokens and padding positions. AutoModelForSequenceClassification adds a classification head to BERT. With num_labels=5, the model returns five logits for each review.

PyTorch: run one prediction

model.eval()
 
example = tokenizer(
    "The food was excellent.",
    return_tensors="pt",
    padding=True,
    truncation=True,
)
 
with torch.no_grad():
    logits = model(**example).logits
 
predicted_class = logits.argmax(dim=-1)
print(tuple(logits.shape))        # (1, 5)
print(predicted_class.item())

model.eval() selects evaluation behaviour for dropout and similar layers. It does not freeze parameters. torch.no_grad() prevents gradient storage during this prediction. The output still contains raw logits, and argmax selects the largest class score.

JJ

Explorer