DistilBERT

DistilBERT is a smaller version of BERT. It uses knowledge distillation to learn from a larger teacher model.

Teacher and student

• Teacher: the larger BERT model
• Student: a smaller Transformer encoder
• Training signal: the teacher's soft labels, or probability distribution over possible outputs

A probability distribution gives information about several possible answers. A hard label identifies one answer.

input text -> BERT teacher -> output probabilities
input text -> smaller student -> output probabilities
                         compare the outputs -> update the student

Size and speed

Property	Reported comparison with BERT
Encoder depth	Half as many encoder layers
Model size	40 percent smaller
Inference speed	60 percent faster
Language-understanding capability	Retains 97 percent

These values describe the reported evaluation. Performance depends on the task and hardware.

Main distinction

DistilBERT reduces model size through teacher-student training. RoBERTa changes BERT's pre-training procedure. DistilRoBERTa uses RoBERTa as its teacher.