Bidirectional Encoder Representations from Transformers (BERT) is a language model that reads text in both directions simultaneously using the Transformer encoder. Unlike a standard language model that predicts the next word left-to-right, BERT can attend to every other word from both sides at once, giving it a much richer understanding of context.
Key Difference from the Original Transformer
| Transformer | BERT | |
|---|---|---|
| Architecture used | Encoder + Decoder | Encoder only |
| Reading direction | Left-to-right (decoder is masked) | Bidirectional |
| Primary task | Sequence-to-sequence (translation) | Understanding / classification |
| Training objective | Next-token prediction | Masked Language Modelling + NSP |
Pre-training Tasks
BERT learns through two self-supervised tasks on unlabelled text.
1. Masked Language Modelling (MLM)
BERT randomly selects 15% of input tokens as prediction targets for Masked Language Modelling (MLM). It predicts their original values from context:
- 80% of selected tokens --> replaced with
[MASK] - 10% of selected tokens --> replaced with a random word
- 10% of selected tokens --> kept as the original word
Input: "[CLS] how are [MASK] doing today [SEP]"
Target: predict "you" at the masked positionThe 80/10/10 split applies to the selected 15% of tokens. The target at every selected position is the original token, including positions kept unchanged or replaced with random tokens.
This forces the model to use bidirectional context rather than just left-to-right patterns.
2. Next Sentence Prediction (NSP)
BERT also learns whether two sentences naturally follow each other through Next Sentence Prediction (NSP):
- 50% of training pairs contain the actual consecutive sentence
- 50% contain a randomly sampled second sentence
[CLS] Sentence A [SEP] Sentence B [SEP]
↓
Binary output: IsNext / NotNext- The
[CLS]token attends to all words in both sentences via self-attention - Its final representation feeds into a binary classifier
- Achieves ~97% accuracy on the NSP task
Tokenization and Fine-tuning
BERT uses WordPiece to split text into subword tokens. Its fine-tuning tasks include:
- Sentence-pair classification, such as semantic similarity
- Single-sentence classification, such as sentiment analysis
- Question answering
- Single-sentence tagging, such as named entity recognition
Autoregressive and Autoencoding Objectives explains the difference between masked pre-training input and normal fine-tuning input.
Special Tokens
| Token | Purpose |
|---|---|
[CLS] | Prepended to every input; its final embedding is used for classification tasks |
[SEP] | Separates sentence A from sentence B; also marks end of input |
[MASK] | Replaces tokens that the model must predict during MLM pre-training |
Why It Works
- Bidirectional attention: every token sees every other token in both directions simultaneously, unlike GPT's left-to-right attention
- Parallelisable: self-attention replaces recurrence, so training is much faster than RNN-based models
- Transfer learning: pre-train once on large unlabelled text, then fine-tune on small labelled datasets for specific tasks (classification, QA, NER, etc.)