RLVR lets the model try a task many times. A program checks each attempt (reward 1 if correct, 0 if not), and the model is trained towards its successful attempts.
Introduction
This is how today's reasoning and agent models are trained, for example GRPO in DeepSeek-R1. The same automatic checks are used to evaluate agents, see Agent Evaluation.
| Piece | Meaning |
|---|---|
| Reward for attempt , for example "did the order stay within the 50 dollar limit" | |
| Grading on a curve: attempts better than the average of the group are reinforced, and worse ones are discouraged | |
| The number of attempts in the group |
Remembering tip
A teacher who grades on a curve. An attempt is rewarded for beating the other attempts of the same group.
Comparison with RLHF
| Point | RLHF | RLVR |
|---|---|---|
| Reward comes from | A reward model trained on human picks | A program that checks the result |
| Best for | Style and what people prefer | Tasks with a checkable answer, such as code, maths and spend limits |
Code
A verifiable reward for the shopping task, and the group advantage:
import numpy as np
def reward(order_total, limit=50.0):
"""1 if the order stayed within the limit, 0 otherwise."""
return 1.0 if order_total <= limit else 0.0
totals = [47.85, 65.18, 47.85, 47.85, 65.18, 47.85] # 6 attempts at the same task
r = np.array([reward(t) for t in totals]) # [1 0 1 1 0 1]
A = (r - r.mean()) / (r.std() + 1e-8) # grading on a curve
print(np.round(A, 2)) # [ 0.71 -1.41 0.71 0.71 -1.41 0.71]Notes:
- attempts above the group average get a positive advantage and are reinforced
1e-8avoids a division by zero when every attempt gets the same reward
After training, the model writes tool calls as JSON, see Tool Call Validation.