Overview
- in-context learning is when a model does a new task from examples placed in the prompt
- no weights change: only the activations change, and no gradient step is taken
- so behaviour is selected by the prompt, not trained, and the ability only shows up in large models
- it ends with zero-shot, one-shot and few-shot, named by how many examples are in the prompt
In-context learning is when a model performs a new task from examples placed in the prompt, without any training.
How it works
- the examples become part of the context that the attention layers read
- only the activations change, because the prompt changes what flows through the network
- no gradient step is taken, and no weight is updated
Why it matters
- one frozen model can be steered to many tasks by the prompt alone
- GPT-3 reported this behaviour as few-shot learning
Prompt as the program
Scale turned the language model into a general-purpose interface that you program with text.
- GPT-3 does a new task from a handful of examples in the prompt, with no gradient update, no task-specific output head and no new parameters
- the wording, the format and the choice of examples select a behaviour that is already latent in the pretrained weights
- the same frozen checkpoint serves translation, question answering, arithmetic and summarisation, and only the text in front of it changes
- so behaviour is selected, not trained: prompt A gives translation, prompt B gives a summary, prompt C solves for x, and the number of gradient steps is 0
- a change costs one sentence instead of one training run, which is why prompting became the default way to use a large model
- in-context learning emerges with scale, so small models do not show it well
A prompt can also unlock step-by-step reasoning, see Chain-of-Thought Prompting.
Important Pointers
- in-context learning is often described as learning, so it is easy to assume gradient descent
- nothing is learned in the parameter sense, and the model is exactly the same before and after
The model that made this behaviour famous is Generative Pre-trained Transformer (GPT).
Note
Three prompt settings are named by how many demonstrations sit in the context.
- zero-shot gives an instruction and no demonstration
- one-shot gives exactly one demonstration
- few-shot gives several demonstrations
All three change activations only, so the count of demonstrations is a property of the prompt and never of the weights.