KV caching stores the key and value vectors of the tokens already generated, so they are not recomputed at every step.
Without it
- at each new token the model would recompute the keys and values of the whole prefix
- the same numbers would be produced again and again, because past tokens never change
Benefit of KV Cache
- each new token computes only its own query, key and value projections
- the cached keys and values of the prefix are read straight from memory
Trade-off
- KV caching does not reduce memory use, it trades compute for memory
- the cache grows with the sequence length and with the number of layers, so a long context makes it large
The step it speeds up is the generation loop of Causal Language Modelling.