A modern instruction following model is built in three stages. Together they form a loop, because failures found at the end feed back to the start.
Stage 1. Pre-training
- objective is next token prediction on raw web-scale text
- data is trillions of unlabelled tokens
- what it gains is knowledge, grammar and latent reasoning
- the gap it leaves is that the model predicts plausible continuations rather than answers
Stage 2. Supervised fine-tuning (SFT)
- objective is cross-entropy on (instruction, response) pairs
- data is curated demonstrations
- what it changes is format, so the model learns to answer rather than continue
- the gap it leaves is that it imitates demonstrations and cannot rank two good answers against each other
Stage 3. Alignment
- we ask the model to generate 2 answers from a single prompt
- we repeat this many times and the human choosen its preferred answer
- in otherwords, data is human preference comparisons of A against B
- 2 methods used are Reinforcement Learning from Human Feedback and Direct Preference Optimization that will score the model and update the weights, etc
The Problem
- however a gap in this is called reward hacking where the model will become relentlessly agreeable (since it scores well) and it produces long padded answers since length correlate with quality, and it collapses onto one safe phrasing for everything
- another gap or issue is drifting from base capability where the model instead gets worse at ordinary task
Solution
- the fix is to add a penalty for moving away from where we started
- now our objective is get a high preference score. and stay close to our SFT version
- KL divergence measures how far we have moved
- we call the preference signal KL-anchored to SFT model
So the three stages change knowledge, then format, then preference.
Loop
- failures found during alignment and evaluation feed back into new SFT data
- eventually they feed back into the next pre-training run
Pretraining objectives by architecture
- encoders and masked language models are pretrained to infill masked words
- decoders and encoder-decoders are pretrained to autoregressively predict text
- for the encoder-decoder family the objective is to give a prefix (first half of sentence) to the encoder without predicting it, then language model will generate the rest
Why a base model behaves oddly
What is given to the model
Serendipity means the occurrence and development of events by chance in a happy or beneficial way. Use the word in a sentence.
What we want from the model
The chance meeting at the bookshop was pure serendipity.
A base GPT-3 answers with more similar instructions
- it was trained to continue text, and a list of similar instructions is a plausible continuation
- so it will return something like
Serendipity can be seen as a lucky accident. Use the word in a sentence. Serendipity is a word that describes ... Use the word in a sentence.
The model this pipeline starts from is Generative Pre-trained Transformer (GPT).
Note
Three behaviours are stacked, and each one is a strictly harder ask than the last.
- generating completions of text is not the same as responding to an instruction
- responding to an instruction is not the same as being aware of social norms and beliefs
- the three stages line up with those three, so pre-training buys the first, supervised fine-tuning buys the second, and alignment buys the third
The encoder branch of the same architecture picture is used differently:
- an encoder turns text into contextualized embeddings
- those embeddings feed classifiers for labeling text, so the natural use is labelling rather than generation