Overview

  • the advantage says whether a choice was better or worse than expected
  • it is reward minus baseline, where the baseline (value) is the expected reward for that situation
  • when raw rewards are all positive, every sampled answer is pushed up; the advantage pushes better-than-expected answers up and worse-than-expected answers down
  • REINFORCE with advantage gives less noisy updates, and it leads into PPO

The advantage answers one question: did this choice turn out better or worse than expected in this situation? Policy gradient methods use the advantage instead of the raw reward, because a raw score alone does not say whether an answer was good.

Definition

where:

  • is the decision being credited
  • is the reward actually received from decision onwards, also called the return
  • is the expected reward before the decision was made, also called the value or the baseline
  • is the situation (state) at decision

For one answer, this is simply . In language-model PPO, often indexes one token step, and implementations usually estimate with a learned value model and future rewards.

Why raw rewards mislead

Case 1, the same score on different prompts:

PromptExpected scoreObserved scoreRaw REINFORCEAdvantageWith advantage
Easy summarisation87pushed uppushed down
Hard reasoning37pushed uppushed up

The reward of 7 is the same, but on the easy prompt it is below normal, so the advantage gives the opposite learning signal.

Case 2, three answers to one prompt, scored on a 1 to 10 scale:

AnswerRaw scoreRaw REINFORCEAdvantage (expected 7)With advantage
A6increaseddecreased
B7increasedno push
C8increasedincreased

With raw scores, all three answers are increased, even answer A, which is below expectation. Answer C gets the biggest push, so on average learning still goes the right way, but it is slow and noisy. With the advantage, the signal is centred on what was expected for this prompt.

What subtracting a baseline does:

  • same direction on average: the gradient still points the same way when averaged over many batches, so the method still learns the same thing (the estimate stays unbiased)
  • less noise: it removes the part of the score that all answers share, which says nothing about which answer was better, so each batch gives a steadier update (lower variance)
  • one condition: the baseline must not depend on which answer was picked, only on the situation, otherwise the direction on average would change

REINFORCE with advantage

where:

  • is the advantage of the -th sampled answer
  • the other symbols are the same as in plain REINFORCE

Five steps:

  1. sample: generate an answer
  2. score: observe the reward
  3. compare: compute the advantage
  4. weight: multiply the log-probability gradient by the advantage
  5. update: move the parameters

What the advantage does to each answer:

  • makes the answer more likely
  • gives the answer no push
  • makes the answer less likely

Note

One part is still missing: a guardrail on how far probabilities can move when the same batch is reused for several updates. That guardrail is Proximal Policy Optimization (PPO).