DistilRoBERTa is a distilled version of RoBERTa. A smaller student model learns to reproduce the behaviour of the larger RoBERTa teacher.
Training process
- Give input text to the RoBERTa teacher
- Use the teacher's output probabilities as a learning signal
- Train the smaller student to produce similar outputs
RoBERTa teacher -> teacher output probabilities -> smaller studentPurpose
- Reduce the size of the model
- Reduce the resources needed to run the model
- Retain useful language-understanding behaviour from the teacher
Distilled model comparison
| Student | Teacher |
|---|---|
| DistilBERT | BERT |
| DistilRoBERTa | RoBERTa |
The teacher determines which model behaviour the student learns. Each student requires its own evaluation of size, speed, and accuracy.