Masked Language Modelling (MLM) trains a model to predict selected tokens from the surrounding text. Bidirectional attention lets the model use context on both sides of a selected position.
BERT masking procedure
BERT selects 15 percent of its input tokens as prediction targets.
| Treatment of a selected token | Share of selected tokens |
|---|---|
Replace it with [MASK] | 80 percent |
| Replace it with a random token | 10 percent |
| Keep its original value | 10 percent |
The 80/10/10 split applies to the selected 15 percent. For every selected position, the target is its original token.
input: [CLS] how are [MASK] doing today [SEP]
target at the selected position: youThe selected positions can include random replacements and unchanged tokens. All selected positions contribute to the token-prediction objective.
Purpose and limitation
- Learn useful representations from unlabelled text
- Use context from both sides to recover missing content
- Support tasks such as classification, question answering, and token tagging
Masked pre-training text differs from the normal text used during fine-tuning. Text generation also requires sequential prediction. See Autoregressive and Autoencoding Objectives.
RoBERTa changes the masking pattern during training. BERT also uses Next Sentence Prediction as a separate pre-training task.