Overview
- GRPO is PPO without the value model
- it samples a group of answers for one prompt, and uses the group's average as the baseline
- each advantage is the reward minus the group mean, divided by the group spread, and a KL term to the frozen reference model is added to the objective
- it comes from DeepSeekMath and is used to train reasoning models such as DeepSeek-R1
Group Relative Policy Optimization (GRPO) is a version of PPO that drops the value network. Instead of learning a model to predict the expected reward, it samples a group of answers for the same prompt and uses the group's own average as the baseline. GRPO was introduced in DeepSeekMath and is used to train reasoning models such as DeepSeek-R1.
My opinion:
GRPO is kinda like finding the z-score, essentially variance of a datapoint but for a group of samples instead of the full dataset. z-score because we are making a comparison of the datapoint against the average that is calculated in place of the value model.
PPO against GRPO

The notation is (the question or prompt) and (the output or answer).
- PPO: the policy writes one output ; the reference model gives a KL penalty, the reward model gives , and a trained value model gives ; GAE combines them into the advantage
- GRPO: the policy writes G outputs for the same question; the reward model scores each one as ; a simple group computation turns them into advantages ; the KL to the reference model is added directly to the objective
In the figure, the value model is the part that disappears. Instead of estimating value with a learned network, GRPO uses the group, which is a non-parametric approach.
Group-normalised advantage
Generate answers and get their rewards. Each advantage is the reward compared with the group:
where:
- is the advantage of answer at token step
- is the reward of answer
- is the mean reward of the answers in the group
- is the standard deviation of the group rewards
- an answer better than the group average gets a positive advantage, and a worse one gets a negative advantage
Example: four answers to one maths question get rewards (correct or wrong). The mean is and the standard deviation is , so the advantages are . The two correct answers are pushed up and the two wrong answers are pushed down, with no value model needed.
When each answer gets one score at the end (outcome supervision), every token of answer gets the same advantage, . The token index only matters with process supervision, where each reasoning step gets its own reward.
The reward itself has no regularisation towards the reference policy, unlike the per-token KL reward in PPO for RLHF. GRPO moves the KL out of the reward and adds it to the objective as a separate term.
GRPO objective
The objective is very similar to PPO:
where:
- is a prompt and are answers sampled from the old policy
- averages over the answers in the group
- averages over the tokens of answer
- the fraction is the probability ratio of the current policy to the old policy for token , as in PPO
- is the group-normalised advantage
- is a penalty for moving too far, with weight
This short form leaves out the clipping. The full GRPO objective in DeepSeekMath applies the and clip of PPO to each token's ratio, in the same way as PPO.
The KL term is estimated a little differently from PPO:
This estimate is always zero or positive, and it is computed per token from the two probabilities only.
KL against the reference model
In the DeepSeekMath paper, the KL term is measured against the frozen reference model (the SFT model), not against :
The two models have different jobs:
- is the snapshot that sampled the current group, and it appears only in the probability ratio, to limit each update
- stays fixed for the whole of training, and the KL term keeps the model close to it, so the model does not drift far from the SFT model, which is the same job as the KL penalty in RLHF
- writing inside the KL is a common simplification, but the reference model is the correct one
Advantage of RL over SFT

On the GSM8K and MATH benchmarks, accuracy is plotted against training steps:
- RFT (offline rejection sampling fine-tuning) stays flat at the lowest level
- online RFT is better
- GRPO with outcome supervision (GRPO+OS) and GRPO with process supervision (GRPO+PS) are the best, and GRPO+PS is highest
Why RL beats this form of SFT:
- RFT is SFT on the model's own correct answers, sampled once from the SFT model before training; every kept answer gets the same weight, and wrong answers are thrown away
- online RFT does the same, but samples from the current model during training, so the data keeps up with the model
- GRPO also samples from the current model, and it also uses the wrong answers: a negative advantage pushes them down, and the size of each push depends on how much better or worse the answer was than the group
DeepSeek-R1

- DeepSeek-R1 reached state-of-the-art results in January 2025, comparable to OpenAI-o1-1217 on AIME 2024, Codeforces, GPQA Diamond, MATH-500, MMLU and SWE-bench Verified
- during RL training of DeepSeek-R1-Zero, its AIME accuracy rises steadily with training steps, and with majority voting over 16 samples it passes the OpenAI-o1-0912 level