A decoding strategy decides which token is actually emitted from the next token distribution. The model is unchanged by this choice, only the output is.
Temperature
- the logits are divided by the temperature before the softmax
- a temperature below 1 makes the distribution sharper, so generation is more conservative
- a temperature above 1 makes it flatter, so generation is more random
Truncation
- greedy decoding always takes the single highest probability token
- top-k keeps a fixed number of candidates
- top-p keeps the smallest set of tokens whose cumulative mass is at least p, so the number of candidates varies from step to step
- beam search keeps a fixed number of whole candidate sequences
Pipeline order
- divide the logits by the temperature
- truncate with top-k or top-p
- renormalise the mass that is left
- sample from what remains
Note
- changing the sampling policy changes the realised output only, and never the learned probabilities, the weights or the tokenizer
- an early sampling error matters because it changes every later conditional distribution and can compound
A worked top-p example:
with and cumulative mass , the third token is the one that first reaches , so the first three tokens are kept.
The softmax step this sits on top of is in Sampling.