Overview

  • in-context learning is when a model does a new task from examples placed in the prompt
  • no weights change: only the activations change, and no gradient step is taken
  • so behaviour is selected by the prompt, not trained, and the ability only shows up in large models
  • it ends with zero-shot, one-shot and few-shot, named by how many examples are in the prompt

In-context learning is when a model performs a new task from examples placed in the prompt, without any training.

How it works

  • the examples become part of the context that the attention layers read
  • only the activations change, because the prompt changes what flows through the network
  • no gradient step is taken, and no weight is updated

Why it matters

  • one frozen model can be steered to many tasks by the prompt alone
  • GPT-3 reported this behaviour as few-shot learning

Prompt as the program

Scale turned the language model into a general-purpose interface that you program with text.

  • GPT-3 does a new task from a handful of examples in the prompt, with no gradient update, no task-specific output head and no new parameters
  • the wording, the format and the choice of examples select a behaviour that is already latent in the pretrained weights
  • the same frozen checkpoint serves translation, question answering, arithmetic and summarisation, and only the text in front of it changes
  • so behaviour is selected, not trained: prompt A gives translation, prompt B gives a summary, prompt C solves for x, and the number of gradient steps is 0
  • a change costs one sentence instead of one training run, which is why prompting became the default way to use a large model
  • in-context learning emerges with scale, so small models do not show it well

A prompt can also unlock step-by-step reasoning, see Chain-of-Thought Prompting.

Important Pointers

  • in-context learning is often described as learning, so it is easy to assume gradient descent
  • nothing is learned in the parameter sense, and the model is exactly the same before and after

The model that made this behaviour famous is Generative Pre-trained Transformer (GPT).

Note

Three prompt settings are named by how many demonstrations sit in the context.

  • zero-shot gives an instruction and no demonstration
  • one-shot gives exactly one demonstration
  • few-shot gives several demonstrations

All three change activations only, so the count of demonstrations is a property of the prompt and never of the weights.