Answers · How models work

How do human pairwise comparisons turn into a reward function?

In shortHuman pairwise preferences become a reward by fitting a scoring model so that preferred responses score higher, then using that score with a penalty for drifting from the starting model.

Fit a scoring function $r_\phi(x,y)$ so that, for two responses to the same prompt, the one people preferred gets the larger score. The comparison is not used as a score directly: it constrains only the difference between two scores, and those constraints train the scoring function.

FIG 1The chance that one response is preferred

: the chance a rater prefers the first response to the second
: the logistic curve, turning an unbounded gap into a probability
: the gap between the two fitted scores, the only thing a comparison can inform

Only the gap between the two fitted scores enters. The logistic curve turns that gap into a probability between 0 and 1, with a gap of 0 giving a coin flip.

The bridge is an assumption: a single underlying score exists, and a person's choice is a noisy readout of the gap between two scores. The logistic curve maps any real gap to a probability. A gap of 0 gives 0.5, a positive gap gives more than 0.5, and a negative gap gives less than 0.5. Fitting chooses $\phi$ to make the observed choices likely. If a rater prefers $y_1$ to $y_2$ , the loss pushes $r_\phi(x,y_1)$ above $r_\phi(x,y_2)$ ; if the rater prefers $y_2$ , it pushes the other way. The push is largest when the model is uncertain, near a gap of 0, and shrinks once the model already predicts the comparison with near certainty.

For one comparison, write the gap as $d=r_\phi(x,y_1)-r_\phi(x,y_2)$ . The model assigns the preferred response probability $1/(1+e^{-d})$ . If $d=2$ , then $e^{-2}=0.1353$ , so the probability is $1/(1+0.1353)=0.8808$ and the loss is $-\log(0.8808)=0.1269$ . If the same pair were ordered the other way, so $d=-1$ , then $e^{1}=2.718$ , the probability is $1/(1+2.718)=0.2689$ , and the loss is $-\log(0.2689)=1.3133$ . The wrong ordering is charged more, and that difference is what moves the scores.

\frac{\partial}{\partial r_\phi(x,y_1)}\left[-\log\sigma(d)\right] = \sigma(d)-1, \qquad \frac{\partial}{\partial r_\phi(x,y_2)}\left[-\log\sigma(d)\right] = 1-\sigma(d).

At $d=2$ , $\sigma(d)=0.8808$ , so the first derivative is $0.8808-1=-0.1192$ and the second is $1-0.8808=0.1192$ . A gradient step raises the preferred response's score and lowers the other's by the same amount. The scoring function is therefore trained on differences, not on an absolute scale.

Only gaps are identified

Add 10 to every score and nothing in the fit changes, because $(r_\phi(x,y_1)+10)-(r_\phi(x,y_2)+10)=r_\phi(x,y_1)-r_\phi(x,y_2)$ . The logistic probability is unchanged. The fitted reward has no zero point and no natural units, so it cannot be compared across runs without care. This is also why a baseline is not optional in the policy gradient: the absolute level of the score carries no information, only its variation across responses does.

Where the natural guess fails

The natural guess is that once $r_\phi$ has been fitted, it is a reward function and can be optimised as hard as you like. That guess fails because $r_\phi$ is accurate only near the responses it was trained on, which look like the starting model's outputs. Far away it is extrapolation. An optimiser does not seek helpful answers; it seeks high scores, and the highest scores in a large response space are disproportionately where the fit is mistaken. A reward model can be 90% accurate on held-out comparisons and still be useless as an optimisation target, because held-out comparisons are drawn from the same narrow distribution. The visible failures are answers that hedge every sentence, restate the question, copy whatever tone the comparisons rewarded, and commit to nothing.

FIG 2Two responses under a fitted reward

	fitted score	divergence from the star	quality judged by people
a direct answer with one	1.2	0.5	8.0
a long hedge that commit	4.8	0.3	2.0

The hedge scores 4.8/1.2=4 times higher on the fitted reward, while human quality is 8.0/2.0=4 times lower. The only in-training signal of trouble is the divergence column, which the tether uses.

The tether

The objective therefore does not use $r_\phi$ alone. It charges the policy for moving away from the model it started from, so the fitted score is worth believing only inside a region.

FIG 3The reward that is actually optimised

: the model training started from, held fixed for the whole run
: how dearly distance is charged, the one knob trading reward against drift
: how much likelier this response became, which is the divergence charged per token

This moves the limit inside the reward. Rather than constraining the step after the fact, the objective itself pays for distance, so the policy stops where the fitted reward stops being worth the drift.

Suppose $\log(\pi_\theta/\pi_{\mathrm{ref}})=2$ , so the new policy makes the response $e^2=7.389$ times likelier. With $\beta=0.5$ the penalty is $0.5\times 2=1.0$ ; a fitted score of 4.0 becomes $R=4.0-1.0=3.0$ . With $\beta=2$ the penalty is $2\times 2=4.0$ and the same response becomes $R=4.0-4.0=0.0$ . The tether does not change the fitted score; it changes how much score is worth paying for in drift.

Where the tether stops helping

If $\beta$ is close to zero the charge is negligible, the policy runs to the fit's errors, and measured reward can rise while human quality falls. If $\beta$ is too large, every response is charged so heavily that the policy barely moves and the fitted reward is never exploited. The tether also assumes the starting model is a sensible anchor: if it is already far from acceptable, the penalty protects the wrong region. And the logistic assumption itself is only an assumption. Human preferences are not always consistent with one underlying score, so some of what $r_\phi$ learns is noise from those inconsistencies, not a stable quality signal.

Worth keeping

A comparison tells you which of two responses is preferred; it never gives an absolute score.
A reward fitted from comparisons is reliable only near the responses it was trained on, so optimising it hard can reward answers that are worse by human judgement.
Penalising divergence from the starting model keeps the policy where the fitted reward still means something.

Where this goes next

This question is one step of Learning From Reward, a course that works through its subject from the beginning. The page that teaches this part of it is When the Reward Is Itself Learned, and it reads the whole way through whether or not you have an account.

What an account changes is the practice: problems standing under every idea, marked with a reason rather than a tick, in a copy of the course that is yours.

Read that page The whole course