Teacher forcing is a training trick. At every position the model is given the ground truth prefix instead of its own prediction.
Why it is done
- the whole correct sequence is already known during training
- every position can therefore be scored in one parallel pass, rather than waiting for the model to generate step by step
Why inference is different
- teacher forcing is not used at inference, because there is no ground truth to feed
- inference feeds back the model's own sampled token
- this gap is why an early mistake at inference can compound through the rest of the sequence
The objective it is used to train is Causal Language Modelling.