RoBERTa means Robustly Optimized BERT Pretraining Approach. It keeps the bidirectional encoder design of BERT and changes its pre-training procedure.
Changes from BERT
| Part | BERT | RoBERTa |
|---|---|---|
| Token prediction | Predicts selected input tokens | Keeps masked token prediction |
| Sentence-pair objective | Uses Next Sentence Prediction | Removes Next Sentence Prediction |
| Masking | Uses masking patterns prepared before training | Changes the masking pattern during training |
| Tokenizer | WordPiece | Byte-level BPE |
| Training data | Original BERT training data | More data and longer sequence segments |
Dynamic masking
- Select 15 percent of input tokens for Masked Language Modelling
- Change the selected positions across training epochs
- Let the model learn from different missing parts of the same text
RoBERTa uses BPE in place of WordPiece. Its encoder can use context on both sides of a token.
Main distinction
Removing Next Sentence Prediction removes one training task. RoBERTa still learns through masked token prediction.
DistilRoBERTa uses RoBERTa as the teacher for a smaller student model.