Attention Masks

A mask tells attention which positions it is not allowed to look at. Two different masks are used, and they do different jobs.

Causal mask

• it stops a position from attending to later positions
• it controls information flow in time, so the model cannot read the future while predicting it
• decoders use it, and encoders do not

Padding mask

• it stops attention to <PAD> positions
• it blocks fake positions that only exist to make every sequence in a batch the same length
• it is paired with ignore_index in the loss, so padded targets are excluded from training

Loss Mask

• decides which positions contribute to the objective, so padded positions and prompt-only positions are excluded
• the padding mask acts on attention, and the loss mask acts on the loss, so one controls what a position may look at and the other controls whether that position is scored
• a prompt-only position is a real token, so the padding mask leaves it alone while the loss mask still removes it from the objective

Important Points

• the two masks are not interchangeable, because one is about time and the other is about fake positions
• encoders can condition on the future and decoders cannot, and the causal mask is the reason to be used for decoders

Both masks are applied to the scores computed in Self-Attention.