In-context learning is when a model performs a new task from examples placed in the prompt, without any training.
How it works
- the examples become part of the context that the attention layers read
- only the activations change, because the prompt changes what flows through the network
- no gradient step is taken, and no weight is updated
Why it matters
- one frozen model can be steered to many tasks by the prompt alone
- GPT-3 reported this behaviour as few-shot learning
Important Pointers
- in-context learning is often described as learning, so it is easy to assume gradient descent
- nothing is learned in the parameter sense, and the model is exactly the same before and after
The model that made this behaviour famous is Generative Pre-trained Transformer (GPT).
Note
Three prompt settings are named by how many demonstrations sit in the context.
- zero-shot gives an instruction and no demonstration
- one-shot gives exactly one demonstration
- few-shot gives several demonstrations
All three change activations only, so the count of demonstrations is a property of the prompt and never of the weights.