Overview
- this note explains why prompting and instruction tuning are not enough, which is the reason for training on human preferences
- prompting: wrong objective, brittle results, nothing saved into the weights, and no way to say one answer is better
- instruction tuning: expensive, many right answers for open tasks, and every token mistake costs the same
- so there is a mismatch between the language-modelling loss and what humans actually want
Prompting and instruction fine-tuning both make a pretrained model more useful, but each has a ceiling. These limits explain why models are also trained directly on human preferences.
Why prompting is not enough
Prompting steers a model that was never trained to be helpful. Post-training changes the model itself.
- the objective is wrong: pretraining maximises the likelihood of internet text, not helpfulness, truthfulness or safety, so a fluent continuation is not the same as an answer
- steering is brittle: results change with the phrasing, the order of examples and the format, and the same question asked twice can get very different answers
- the fix does not persist: nothing learned at inference time is written back into the weights, so every user has to find the right prompt again
- no way to express preference: given two fluent answers, prompting gives no signal that one is better than the other
| Approach | What it does | Weights |
|---|---|---|
| Prompting | Steers the model at inference time | Unchanged |
| Post-training | Supervised fine-tuning, then RLHF, DPO or GRPO | Actually change |
Limits of instruction fine-tuning
Instruction fine-tuning (supervised fine-tuning on instruction and answer pairs) changes the weights, but it still has limits.
- it is expensive: ground-truth answers must be collected for every task
- problem 1, no single right answer: open-ended creative tasks, such as "Write me a story about a dog and her pet grasshopper", have many good answers, so imitating one answer is a weak signal
- problem 2, all mistakes cost the same: the language-modelling loss penalises every token-level mistake equally, but some mistakes are much worse than others

In the figure, the target is "Avatar is a fantasy TV show". Predicting "adventure" instead of "fantasy" is a small error in meaning, and predicting "musical" instead of "show" is a bigger error. The loss treats both as one wrong token.
So even after instruction fine-tuning, there is a mismatch between the language-modelling objective and the real goal, which is to satisfy human preferences. This leads to one question: can the model be trained to satisfy human preferences explicitly?
Note
A simple first step is to sample several answers and train only on the best ones, described in Rejection Sampling Fine-Tuning.