Overview
- rejection sampling fine-tuning (best-of-N) is the simplest step towards preference training
- the model writes N answers, a reward model scores them, and the model is trained only on the best ones
- offline version samples from the fixed SFT model, and the online version samples from the model being trained
- it only keeps or drops answers, which leads to methods that use the reward score directly
Rejection sampling fine-tuning, also called best-of-N, is a simple way to make better use of the same model. The model generates N answers, a scorer picks the best ones, and the model is trained to make those best answers more likely.
How it works

- give a single prompt to the policy language model
- generate a batch of N completions
- pass every completion through a reward (preference) model, which gives each one a score
- keep the best 1 to K completions (the "top data") and reject the rest
- train the model to maximise the likelihood of the top data
Objective
The supervised fine-tuned policy is trained to maximise the likelihood of the top data:
where:
- is the training objective for rejection sampling fine-tuning, which is maximised
- is the set of model parameters that are being trained
- is a prompt taken from the prompt dataset
- is a full answer sampled from the supervised model for prompt
- is the number of tokens in answer , so averages over tokens
- is an indicator that equals 1 when is one of the best-scored answers and 0 otherwise, so rejected answers add nothing
- is the -th token of the answer, and is all tokens before it
- is the probability that the model being trained gives to token
Training uses gradient ascent, because the objective is maximised.
Offline and online versions
- offline: the answers are sampled from the fixed supervised model,
- online rejection sampling fine-tuning: the answers are sampled from the current model that is being trained, , so the data improves as the model improves
My way of understanding:
Their difference is just the timing of generation of samples.
For offline rejection sampling fine-tuning, we are asking the starting model to generate all the samples first, and then we use the reward model (or the preference model) to score it, etc.
But for online rejection sampling fine tuning, we are asking the model to generate the samples on the fly. Meaning the current model generates the samples, we grade it with the reward model, weights change, and the new model generates samples again, and the cycle repeats.
Note
Rejection sampling only keeps or drops answers. A more direct method uses the reward score itself to push the model, see Policy Gradient and REINFORCE.