Bidirectional Encoder Representations from Transformers (BERT) is a language model that reads text in both directions simultaneously using the Transformer encoder. Unlike a standard language model that predicts the next word left-to-right, BERT can attend to every other word from both sides at once, giving it a much richer understanding of context.

Key Difference from the Original Transformer

TransformerBERT
Architecture usedEncoder + DecoderEncoder only
Reading directionLeft-to-right (decoder is masked)Bidirectional
Primary taskSequence-to-sequence (translation)Understanding / classification
Training objectiveNext-token predictionMasked Language Modelling + NSP

Pre-training Tasks

BERT learns through two self-supervised tasks on unlabelled text.

1. Masked Language Modelling (MLM)

BERT randomly selects 15% of input tokens as prediction targets for Masked Language Modelling (MLM). It predicts their original values from context:

  • 80% of selected tokens --> replaced with [MASK]
  • 10% of selected tokens --> replaced with a random word
  • 10% of selected tokens --> kept as the original word
Input:  "[CLS] how are [MASK] doing today [SEP]"
Target: predict "you" at the masked position

The 80/10/10 split applies to the selected 15% of tokens. The target at every selected position is the original token, including positions kept unchanged or replaced with random tokens.

This forces the model to use bidirectional context rather than just left-to-right patterns.

2. Next Sentence Prediction (NSP)

BERT also learns whether two sentences naturally follow each other through Next Sentence Prediction (NSP):

  • 50% of training pairs contain the actual consecutive sentence
  • 50% contain a randomly sampled second sentence
[CLS] Sentence A [SEP] Sentence B [SEP]
                                   ↓
                    Binary output: IsNext / NotNext
  • The [CLS] token attends to all words in both sentences via self-attention
  • Its final representation feeds into a binary classifier
  • Achieves ~97% accuracy on the NSP task

Tokenization and Fine-tuning

BERT uses WordPiece to split text into subword tokens. Its fine-tuning tasks include:

  • Sentence-pair classification, such as semantic similarity
  • Single-sentence classification, such as sentiment analysis
  • Question answering
  • Single-sentence tagging, such as named entity recognition

Autoregressive and Autoencoding Objectives explains the difference between masked pre-training input and normal fine-tuning input.

Special Tokens

TokenPurpose
[CLS]Prepended to every input; its final embedding is used for classification tasks
[SEP]Separates sentence A from sentence B; also marks end of input
[MASK]Replaces tokens that the model must predict during MLM pre-training

Why It Works

  • Bidirectional attention: every token sees every other token in both directions simultaneously, unlike GPT's left-to-right attention
  • Parallelisable: self-attention replaces recurrence, so training is much faster than RNN-based models
  • Transfer learning: pre-train once on large unlabelled text, then fine-tune on small labelled datasets for specific tasks (classification, QA, NER, etc.)