Overview

  • this note explains why prompting and instruction tuning are not enough, which is the reason for training on human preferences
  • prompting: wrong objective, brittle results, nothing saved into the weights, and no way to say one answer is better
  • instruction tuning: expensive, many right answers for open tasks, and every token mistake costs the same
  • so there is a mismatch between the language-modelling loss and what humans actually want

Prompting and instruction fine-tuning both make a pretrained model more useful, but each has a ceiling. These limits explain why models are also trained directly on human preferences.

Why prompting is not enough

Prompting steers a model that was never trained to be helpful. Post-training changes the model itself.

  • the objective is wrong: pretraining maximises the likelihood of internet text, not helpfulness, truthfulness or safety, so a fluent continuation is not the same as an answer
  • steering is brittle: results change with the phrasing, the order of examples and the format, and the same question asked twice can get very different answers
  • the fix does not persist: nothing learned at inference time is written back into the weights, so every user has to find the right prompt again
  • no way to express preference: given two fluent answers, prompting gives no signal that one is better than the other
ApproachWhat it doesWeights
PromptingSteers the model at inference timeUnchanged
Post-trainingSupervised fine-tuning, then RLHF, DPO or GRPOActually change

Limits of instruction fine-tuning

Instruction fine-tuning (supervised fine-tuning on instruction and answer pairs) changes the weights, but it still has limits.

  • it is expensive: ground-truth answers must be collected for every task
  • problem 1, no single right answer: open-ended creative tasks, such as "Write me a story about a dog and her pet grasshopper", have many good answers, so imitating one answer is a weak signal
  • problem 2, all mistakes cost the same: the language-modelling loss penalises every token-level mistake equally, but some mistakes are much worse than others

In the figure, the target is "Avatar is a fantasy TV show". Predicting "adventure" instead of "fantasy" is a small error in meaning, and predicting "musical" instead of "show" is a bigger error. The loss treats both as one wrong token.

So even after instruction fine-tuning, there is a mismatch between the language-modelling objective and the real goal, which is to satisfy human preferences. This leads to one question: can the model be trained to satisfy human preferences explicitly?

Note

A simple first step is to sample several answers and train only on the best ones, described in Rejection Sampling Fine-Tuning.