Causal Language Modelling

Causal language modelling is the training objective of a decoder. The model reads a prefix and predicts the next token, one position at a time.

Shifted Training Pair

The input and the target are the same sequence, offset by one position:

• the input drops the last token, written x = batch[:, :-1]
• the target drops the first token, written y = batch[:, 1:]
• position i of the input therefore lines up with the token that follows it in the target

Loss

• the loss is cross-entropy over the vocabulary, so every position is a classification over V classes
• in logits.reshape(-1, V), V is the number of token classes in the vocabulary

Cost

• the attention scores have shape B x h x T x T, so they are quadratic in context length
• doubling the context length therefore quadruples the dense attention score work

The model trained with this objective is Generative Pre-trained Transformer (GPT).