Overview
- DPO trains directly on preference pairs, with no separate reward model and no RL
- key insight: the RLHF goal has a closed-form best answer, so the model itself can act as the reward
- the loss raises the winner's likelihood and lowers the loser's, both measured against a reference model
- it is simpler than PPO and does as well or better, and the idea also works outside text
Direct Preference Optimization (DPO) trains a language model directly on preference pairs, without a separate reward model and without reinforcement learning. It replaces the complex RL part of RLHF with a simple weighted maximum-likelihood objective.
In plain words, "weighted maximum likelihood" means DPO raises the likelihood of the winning answer and lowers the likelihood of the losing answer, like SFT on the winner and the opposite on the loser. Each pair gets a weight that is large when the model still ranks that pair the wrong way round.
DPO is offline. It trains on a fixed set of preference pairs and does not sample new answers during training, which is a large part of why it is cheap. The cost is that it never learns from its own new answers.
In one picture

- RLHF: preference data trains a reward model by maximum likelihood, then reinforcement learning samples completions, labels them with rewards and updates the LM policy
- DPO: preference data trains the final LM directly by maximum likelihood
Why remove RL
RL with PPO can be quite complex:
- it is computationally expensive and tricky
- a value function must be fitted
- online sampling is slow, because the model must generate new answers during training
- performance can be sensitive to hyperparameters

The figure shows how many parts a real RLHF and PPO workflow needs: an SFT model, a reward model, a value model, a policy, GAE, an experience buffer and several losses.
Key insight: closed-form solution
RLHF maximises the reward while staying close to the pretrained model:
where:
- is the prompt and is a sampled answer
- is the model being trained
- is the starting (pretrained or SFT) model, also called the reference model
- is the reward-model score
- is the weight of the KL penalty
This objective has a closed-form solution, which is the best possible model :
where:
- is the optimal model
- is a normalising constant that makes the probabilities sum to 1, and it depends only on the prompt
The best model is the reference model, reweighted so that high-reward answers become more likely.
Turning the model into a reward
Take the log of both sides and rearrange:
This holds for any language model, so the reward can be written using the model being trained:
So the language model itself defines a reward. There is no need to train a separate reward model.
DPO loss
Insert this derived reward into the Bradley-Terry model of human preferences:
where:
- is the dataset of preference triples
- is the winning (preferred) answer and is the losing answer
- the first log ratio is the implicit reward for the winning sample, and the second is the implicit reward for the losing sample
- is the sigmoid function
- has the same role as the KL weight in RLHF: a larger keeps the model closer to the reference model
The term cancels, because the loss only measures the difference between two rewards for the same prompt.
What the gradient does
The gradient of the DPO loss is:
with the implicit reward , where:
- increases the likelihood of the winner
- decreases the likelihood of the loser
- gives a higher weight when the reward estimate is wrong, that is, when the model still ranks the loser above the winner
Results

On TL;DR summarisation, win rates against human-written summaries are judged by GPT-4 at different sampling temperatures. DPO has the highest win rate at most temperatures. PPO falls sharply at high temperatures. Best of 128 sampling is competitive but expensive, and SFT, Preferred-FT and GPT-J are lower.
Other variants of this idea, such as KTO and IPO, are also emerging.
Note
The DPO objective never mentions language. It only needs a winner, a loser and a reference model, so it also works for other modalities, see Tango 2.