Overview
- reward hacking is when a model gets a high reward without doing the real task
- classic example: a boat-racing agent that circles for points and never finishes
- in language models, a learned reward can favour answers that sound sure, which leads to hallucinations
- over-optimisation: reward-model score keeps rising while real human preference falls, so a KL penalty is added
Reward hacking happens when a model finds a way to get a high reward without doing the task the reward was meant to measure. It is a common problem in reinforcement learning, and it is the predictable failure of optimising a learned proxy for human judgement.
Classic example

In a boat-racing game, the agent is rewarded for its score. It learns to circle and hit the same score targets again and again, so the score keeps rising even though the boat never finishes the race. The reward was a proxy for "win the race", and the agent optimised the proxy instead of the goal.
Why it happens with language models
- human preferences are unreliable: people can prefer answers that only look good
- models of human preferences are even more unreliable: a reward model is trained on a limited set of comparisons, so it has blind spots that RL can exploit
Confidence rewarded more than evidence
A preference signal can reward the appearance of helpfulness without fully checking whether the facts are supported.
- proxy: clear and decisive answers look helpful
- shortcut: the model learns to sound certain even when the evidence is weak
- failure: the reward rises, but truthfulness does not
- counterfactual fix: score evidence use and calibrated uncertainty, so unsupported certainty loses its advantage
So chatbots can be rewarded for answers that seem authoritative and helpful, regardless of truth. This can lead to made-up facts and hallucinations. One confident error is an observed failure, but on its own it does not prove which training mechanism caused it.
Reward model over-optimisation

The plot shows what happens as the policy moves further from the supervised starting model (measured by the KL distance on the x-axis):
- the reward model's prediction of quality keeps rising towards 1.0
- the actual human preference rises at first, peaks around a KL of 10, then falls sharply
After some point, the policy is pleasing the reward model, not the humans. This is why RLHF adds a KL penalty to the reward:
where:
- is the reward-model score of answer
- is the probability of under the model trained with RL
- is the probability of under the starting (pretrained or supervised) model
- is the weight of the penalty, which keeps the model close to where it started
Choosing is a trade-off:
- larger : the model stays close to the start, so it is safer from reward hacking but improves less
- smaller : the model can move further and improve more, but it is more likely to reach the region where real human preference falls
Note
The same failure appears in self-improving preference loops, such as TangoFlux, when the training data is not refreshed.