This is the go-to intrinsic metric for comparing language models. It is able to directly measure how well the model's learned probability distribution matches the real data distribution.
NLL stands for negative log-likelihood. NLL is the average "surprise" of the model at each correct token. Surprise here is a information theory term that is formally called self-information.
For every token in the sequence, we take the actual output from the model at every next token prediction, taking log and averaging the negatives over all N tokens. A model that is confident and wrong (or unsure) gets a large NLL.
Reading perplexity as a branching factor
Perplexity behaves like an effective branching factor, which is the number of equally likely choices the model still faces at each next token decision.
- a model that puts probability 1 on the correct token at every step has a perplexity of 1, because
- a model that spreads probability evenly over 3 options has a perplexity of 3, because 3 choices remain equally open
- lower perplexity therefore means the model needs fewer effective choices per decision
What perplexity cannot tell you
- perplexity is intrinsic, so it says which model assigns higher probability to unseen text
- it does not say which model is better for a specific downstream task
- that claim needs a task based number such as Word Error Rate (WER)