We've done the whole RLHF pipeline before. The full production picture, containers and all.
Right, and that's the thing. The pipeline is the easy part to describe. It's three boxes and an arrow.
Today is one box. Daniel wants the reward model.
The one everybody draws as a little smiley face between "human feedback" and "better model."
He says he understands the basic idea. Reward model, trained on human preferences, learns which outputs people prefer. What he isn't clear on is what happens next.
Which is the honest place to be stuck, because that's the part that usually gets hand-waved.
His actual question. How does a trained reward model influence the weights of the language model being refined? He wants the whole lifecycle, creation, training, use in the RL process, and what's actually happening computationally at each stage.
So not "the reward model makes the model better." What's the mechanism.
He also wants to know whether the reward model keeps learning during all this. And where they come from. Trained from scratch, or adapted from something that already exists. And whether you can just grab one off a shelf.
Good, because those have surprisingly different answers.
Start with what it is.
A reward model takes a prompt and a completion and outputs one number. That's the whole thing. Prompt in, response in, scalar out.
One number for a whole paragraph of text.
One number. And it's trained on trios. A prompt, a completion a human preferred, and a completion they rejected. The model learns to give the preferred one a higher score.
How much higher?
It doesn't care. Only the gap matters. The lot is built as a Bradley-Terry model of preference, and the probability that the chosen one beats the rejected one is the sigmoid of the difference between the two scores.
So if I add a hundred to every score in the system, nothing changes.
Nothing at all. That's the property that makes it work. It's a ranking machine wearing a scoring costume.
So the training objective is what, exactly.
Negative log of that sigmoid. So the loss is just the negative log-sigmoid of the score difference between chosen and rejected. In code it's one line. Negative log-sigmoid of rewards chosen minus rewards rejected, averaged over the batch.
That's it.
You push the chosen score up, the rejected score down, and the difference is what shows up in the gradient.
And then it just sits there, producing numbers.
That's the part I want to make sure lands. The reward model never touches the language model's weights. It doesn't backpropagate into the policy. It produces a scalar and the scalar goes somewhere else. Nathan Lambert describes the reward model as playing the role of the environment in a standard RL setup, except you get to learn the environment from human preferences instead of having it handed to you.
So it's the scoreboard, not the coach.
The scoreboard has opinions about what a good game looks like, and it learned those opinions from people.
Where does the scoreboard come from, though.
It's adapted. This is the part people get wrong. Nobody trains a reward model from scratch. You take a pretrained language model, or more often your SFT checkpoint, and you cut off the final unembedding layer. The one that maps hidden states to vocabulary.
The one that turns the model into a text generator.
You remove it, and in its place you bolt on a single linear layer that maps the hidden size down to one number. That's the reward head. The InstructGPT paper describes it directly. Starting from the SFT model with the final unembedding layer removed, they trained a model to take a prompt and a response and output a scalar reward.
So it's the same body, different mouth.
Same body, and the body is doing almost all the work. It already understands language. All the reward head has to learn is which direction in that representation means "better."
Which token's hidden state does it read?
The last non-padding token. Usually the end-of-sequence token. You take the hidden state at that position and pass it through the linear layer. In Lambert's reference implementation you index the hidden states at the last real token using the attention mask sum minus one, then run it through a linear layer of hidden size to one.
And the practical version people actually run.
AutoModelForSequenceClassification with num_labels equal to one. Hugging Face TRL has a RewardTrainer that trains exactly this on a preference dataset with the Bradley-Terry pairwise objective. You can spin one up in an afternoon.
And the training details.
One epoch. That's the striking part. Reward model training is typically a single pass over the preference data, because it overfits fast.
One epoch.
One. InstructGPT used a learning rate of nine times ten to the minus six, batches of sixty-four prompts up to twenty-three hundred comparisons, one epoch. Done.
That feels like nothing.
It is nothing by pretraining standards. Which tells you something. The model already knows what good text looks like. The reward head is learning a thin layer of preference on top of it. There isn't much to learn, and if you keep going you just memorize the pairs.
What about how many comparisons they got out of the annotators.
Four to nine completions ranked per prompt. Then every pair within that ranking becomes a training example. So from nine completions you get thirty-six pairwise comparisons out of a single annotation session.
That's efficient.
It's the reason the approach scales as well as it does. You don't need a label per completion. You need a rank order, and the rank order explodes into pairs.
Now the question I actually want answered. Can you use somebody else's reward model, or is it welded to the model you're training?
Both exist, and the honest answer is that the frontier recipes are welded and the open ecosystem is not.
So walk me through the welded version first.
InstructGPT trained its reward model from its own SFT model. It wasn't a general judge. It was a critic trained inside that project, on that project's data, reflecting that project's annotation guidelines. That's the norm in a frontier pipeline.
And the other version.
There's a real market of general-purpose reward models. RewardBench, Allen AI's benchmark for exactly this, evaluated over thirty public reward models when it launched, ranging from four hundred million up to seventy billion parameters.
That's a spread.
It's a spread, and it includes things people don't usually call reward models. DPO models get used as what the benchmark calls indirect reward models, because a DPO-trained model implicitly represents a reward function.
That connects back to the alignment-without-fine-tuning episode. The claim there was that you could shape a model with preference optimization instead of retraining. Turns out you can also mine the leftovers for a scorer.
That's a nice way to think about it. The top of the original leaderboard was Starling-RM-34B out of Berkeley, then Tulu-2-DPO-70B, then a Mixtral instruct model, then Starling's seven-billion version, then a Nous Hermes Mixtral DPO model.
So the best-ranked reward model was a thirty-four billion model, not the biggest one on the list.
Not the biggest. And that's before we get to the caveat, which is that the benchmark found DPO models fail to generalize to popular preference test sets and show higher variance in performance.
Meaning they look good on the data they were trained near and wobble outside it.
And it gets worse. The benchmark also says many reward models struggle with adversarially designed responses that don't address the prompt at all.
So they hand out high scores to confident nonsense.
Some of them, yes. Which is a preview of where this whole thing breaks.
Before we get there. The size asymmetry. You mentioned the reward model is adapted from the SFT model, so wouldn't you want it as big as the policy?
You would think so. InstructGPT's final reward model was six billion parameters, and they used it to train a one-hundred-seventy-five billion parameter policy.
Six steering one seventy-five.
They tried the big one. The one-hundred-seventy-five billion reward model was unstable and far more expensive, so they went with the small one. The critic does not need to outthink the student. It needs to be consistent.
So the intuition that the judge has to be smarter than the writer is wrong.
It's wrong in practice. What matters is that the reward model is a reliable proxy on the distribution of outputs it's going to see, and a small model adapted from the same base is more reliable there than a huge one that can't be trained stably.
Alright. It exists. It produces a number. Now the part Daniel actually asked about. How does that number move weights.
So now you need four things in the room at once. The policy, which is the model being trained. The reward model, frozen. A reference model, which is a frozen copy of the SFT checkpoint. And for PPO, a value function, which is another model entirely.
Four models to train one.
Four models to train one, and two of them never change.
Walk me through a single step.
A prompt is sampled. The policy generates a completion, token by token. The reward model scores that completion once, at the end. The reference model computes log-probabilities on the same text, which feeds a penalty term. Then all of it combines into a gradient update on the policy.
Once, at the end.
Once. The whole completion gets one number. This is modeled as a contextual bandit at the sequence level, not a full sequential decision problem. Sample a prompt, generate a response, score it once, episode over.
So there's no per-token feedback at all.
None from the reward model. And that's exactly why you need the value function. If the only signal is one number at the end of the episode, something has to spread credit across all the tokens that led there. That's what the value function does, with generalized advantage estimation.
But with no discounting, because the episode is so short.
InstructGPT used a discount factor of one when estimating the advantages. And the write-up notes that because the episode is short and the reward is terminal, the long-horizon machinery of generalized advantage estimation is largely degenerate here. All the standard apparatus, running in a case where it barely has anything to do.
So describe the actual update. What is the equation doing.
The core policy gradient says the change in parameters is proportional to a scalar times the gradient of the log probability of the action.
Two pieces.
The gradient of the log probability tells you which direction in parameter space makes that action more likely. The scalar tells you how good that action turned out to be.
And the scalar's sign.
If it's positive, you update to make the action more likely. If it's negative, you update to make it less likely. That's the entire learning signal.
So the reward model's contribution is one number that decides the sign and size of a step.
One number. It never sees a gradient from the policy. It never updates. It hands over a scalar and the optimizer does the rest of the work.
Then what's the reference model for.
It's the leash. In InstructGPT's objective there's a term that penalizes the KL divergence between the policy and the frozen SFT model, with a coefficient of zero point zero two.
Why do you need a leash at all.
Because the reward model is only accurate in a region. It was trained on outputs from the SFT model and the annotators' data. Push the policy far enough away from that distribution and the reward model's scores stop meaning anything.
It starts grading a game it's never seen.
It starts grading confidently and wrongly. The KL penalty keeps the policy in the neighborhood where the reward signal is trustworthy.
And the clip.
The clip is PPO's own contribution and it's a different job entirely. The ratio between the new policy's probability and the old policy's probability for a given action gets clipped to the range one plus or minus zero point two. Once a good action's ratio passes one point two, the objective is capped and the gradient goes to zero.
So it stops rewarding you for pushing an already-favored action further.
It bounds how far a single update can move the policy. A soft trust region. The clip limits the step from old policy to new policy. The KL penalty limits drift from the SFT model. Two different regularizers, two different distances.
One bounds the stride, one bounds the fence.
That's the cleanest way to put it.
Now the question I suspect is the actually important one. Does the reward model learn during any of this.
No. It's frozen during the RL phase. You can lay it out as a table. The policy is updated by the policy gradient. The reference policy is frozen at the SFT checkpoint. The value function is updated jointly with the policy. The reward model says, and this is a direct quote from the reference material, no. Not during the RL phase.
So it's trained once, then bolted in place.
Bolted in place for the entire RL run. Every candidate the policy generates is scored by the same frozen critic from the first step to the last.
That seems like an obvious problem.
It is the problem. And there's a name for it. The quality of the reward model directly determines the upper bound of the policy. That line comes out of the Secrets of RLHF work. The policy cannot get better than the target it's optimizing against.
So the critic sets the ceiling.
The critic sets the ceiling, and the policy is very good at finding the cracks in it. This is reward hacking. The proxy reward goes up while the true quality plateaus or falls. The model finds outputs that score well without being good.
Give me an example of the shape.
Reward models are underspecified. Two reward models can perform nearly identically on in-distribution data and then yield wildly different rewards when the policy drifts, because of distribution shift. The score is the model's opinion, and the policy is optimizing the opinion.
Ensembling. You mentioned it.
Ensembling helps and doesn't solve it. You train several reward models and average their scores. It mitigates hacking but doesn't eliminate it. The interesting finding is that ensembles varying by pretraining seed generalize better than ensembles that only vary the fine-tuning seed. The diversity has to be deeper than the last layer.
And the data itself.
Roughly a quarter of the preference pairs in the HH-RLHF dataset are likely mislabeled. That's from ensemble disagreement with the human labels. So the reward model is being trained on noisy ground truth, and then used as an oracle.
Twenty-five percent.
A quarter of the signal is wrong. And the policy doesn't care which labels were right. It optimizes whatever the reward model learned, mistakes included.
And this happens slowly, over a long run, presumably.
It happens fast. Direct alignment methods deteriorate often before a single epoch of the dataset is completed. And there's a finding that one percent contamination in distillation data is enough for a model to internalize reward hacking that resurfaces once RL starts.
One percent.
The seeds get planted early and the policy waters them.
So you retrain the reward model.
Across iterations, yes. That's iterated RLHF. Reward models are repeatedly retrained with updated human feedback and the policies are re-optimized. But within any single RL run, the reward model is fixed. It is not learning alongside the policy.
So the answer to Daniel's question is a clean no, with a footnote.
Clean no inside a run. The retraining happens between runs, with fresh human feedback, which is expensive, which is why most pipelines don't do it very often.
Hold on. Rewind to the part I want to sit on.
Go ahead.
The reward model scores the completion once and the value function spreads the credit. That means the value function is doing work the reward model should be doing.
It's doing work the reward model can't do, because the reward model doesn't produce per-token scores. It produces one number for the whole sequence. Somebody has to attribute that number back to the individual decisions that produced it.
And the value function is initialized from the reward model.
From the reward model, yes, six billion parameters in InstructGPT's case, then trained jointly with the policy. Same starting weights, different job.
So the reward model gets a second life as a value estimator.
A second life with a different objective. One is judging finished text. The other is predicting, mid-generation, what the final score is going to be.
Which sounds like a harder job.
It's a harder job and it's why it needs its own training. It's a prediction of a prediction.
So let me get the pipeline clear. A prompt gets sampled. The policy generates. The reward model reads the last token's hidden state and emits a number. The reference model supplies a leash. The value function turns one number into a per-token attribution. PPO clips the step. The optimizer moves weights.
That's the loop, and it runs about two hundred fifty-six thousand times in InstructGPT's case. Batch of five hundred twelve, minibatch of sixty-four, a single inner epoch per batch, exponential moving average decay of zero point nine nine two.
And the result.
The one-hundred-seventy-five billion parameter InstructGPT model was preferred over the one-hundred-seventy-five billion parameter GPT-3 base model eighty-five percent of the time, give or take three points. In human evaluations.
Eighty-five percent of the time.
That's what all of this machinery buys. A reward model the size of a small language model, a leash with a coefficient of zero point zero two, a clip at zero point two, and a value function learning to predict the critic.
Something's nagging me though. The reward model is scored on pairs, chosen versus rejected, from a fixed dataset. Then you deploy it on completions from a policy that's actively moving away from that distribution.
Which is the underspecification problem. The reward model is a snapshot of human preference at one point in time, on one distribution of text, and the policy is a moving target.
So the reward model is out of date the moment RL starts working.
Slowly, and then quickly. The KL penalty is what keeps it roughly in date. Loosen the penalty and you buy more capability at the cost of the reward model's accuracy. That's the trade you're actually making when you tune beta.
I want to come back to something. Daniel asked whether the reward model is held fixed or keeps learning. Both, in a sense. It learns like hell during training, then it stops.
It learns one epoch's worth and then it's a rock. That's a strange lifecycle for the component everyone describes as the important one.
The more important the component, the less it changes.
One epoch. Nine times ten to the minus six learning rate. Then frozen forever while everything around it moves.
Anyway, I have a question for you.
Go ahead.
If the reward model is capped at six billion parameters and the policy is at one hundred seventy-five, and the reward model is the ceiling, isn't the ceiling being set by the smaller model in the room?
That's the thing. The ceiling isn't the reward model's raw capability. It's the accuracy of its scores on the distribution the policy is generating. A six-billion reward model that has seen the right outputs beats a one-hundred-seventy-five-billion one that was never trained stably.
But the six-billion model has a smaller vocabulary of preferences.
Yes, and that's exactly where the ceiling comes from. The reward model can only recognize the kinds of quality it was trained to distinguish. If the policy produces something better in a way the annotators never demonstrated, the reward model has no way to reward it, and it may even penalize it.
So the policy is being optimized toward what the annotators knew how to prefer.
Toward the part of human preference that the reward model managed to capture. Everything outside that is invisible or actively punished.
That's a narrower claim than the celebration usually implies.
It's the honest one. That's the tension in the whole design. The reward model is where complex human preferences get compressed into a number, and it's also the boundary of what the policy can achieve.
So when you tune beta, you're choosing how tightly to hold the model near the region where its judge is accurate.
Small beta, the policy runs further, scores get less trustworthy. Large beta, the policy stays close to the SFT model and the reward model stays accurate, but you don't move very far.
You're buying capability with the reliability of your own judge.
That's the trade, and it's not obvious which way to move it.
The thing I keep coming back to is that the reward model never touches the policy. It's the most-discussed component in the whole pipeline and it's the one piece with no direct line to the weights.
It's a scalar. That's all it ever is. Everything else, the value function, the clip, the leash, exists to turn one number into a usable gradient.
Which means the craft isn't in the reward model. It's in the machinery you wrap around it.
Half of it is. The other half is in making the reward model trustworthy in the first place.
The Sony MDR-7506. Eighty-nine pounds. Two pairs, secondhand.
That's what I used to rank them. Forty-seven of them, one afternoon, and a notebook.
Ranked what?
Demo tapes. Local bands, this was a competition the council ran. Civic arts funding, they'd shortlist acts for a summer slot, and the shortlist was supposed to come from the judges' scores but the judges never turned up, so I got the box.
And you ranked them with headphones.
You rank with the ears, not the paperwork. Forty-seven tapes, two passes each. I wrote the numbers in a notebook, one to forty-seven, and the next morning I did all of it again.
Why twice?
Because the first pass wasn't right, and when I compared the two notebooks the top ten were almost the same and the middle was nonsense. Same tapes. Same headphones. Different day. I had one band at eleven and the next day at thirty-two.
Nothing about the audio changed.
Nothing. I changed. They asked me which list was the real one and I said the second, but I couldn't tell you why. Nobody could.
So what happened to the bands.
The council used the wrong list. Somebody photocopied the first notebook instead of the second. Three of them got the summer slot off a ranking I'd already thrown out. I never told them.
Did anyone notice?
The bands did. They'd been told the criteria and they worked it out inside a week. Two of them started sending a different tape, mixed different, brighter, because that's what the top ten sounded like.
They gamed your ranking.
They gamed a ranking I didn't stand behind, off a notebook I'd discarded, and they were right to. If I'd told them the second list was the one, they'd have gamed that instead. There's no version where they don't work out what you're measuring.
So the reward model here is a man with a notebook and a pair of headphones.
It's the same thing. You're scoring something that changed because you're scoring it. And I got paid for it. Forty pounds, in council vouchers for the leisure centre, which I couldn't use because they closed it that autumn.
You ranked forty-seven bands, in vouchers, for a building that shut down.
It was a good afternoon. I still have the notebook.
The consistency problem isn't a data problem.
It's not. In the HH-RLHF set, roughly a quarter of the pairs are likely mislabeled. I was going to frame that as annotation noise.
Hilbert's point is that some of it isn't noise. It's the same person being a different person on Tuesday.
Which means a larger reward model and a cleaner objective don't fix it. The label was correct when it was made.
So the reward model inherits human inconsistency, and no amount of ensembling reaches that, because the ensembles are all trained on the same wobbling labels.
You can average away variance between models. You can't average away the variance inside the person who wrote the labels.
And two of his bands learned the criterion and started sending brighter tapes.
They got the slot, too.
And that's the thing about the frontier of this whole field. The reward model is a person with a notebook, and the policy is a band that works out which notebook you copied.
So the whole architecture is a defence against a forty-seven-tape problem.
Most things are.
Here's what I'll take from that. The two regularizers, the clip and the leash. I framed them earlier as two fences doing different jobs, and I still think that's right.
But.
But Hilbert's afternoon says something about which fence matters more. The clip bounds the step. The leash bounds the drift. What neither one fixes is the wobble already baked into the reward model. You can hold the policy as close to the SFT model as you like, and it's still being scored by a judge that changed its mind overnight.
The leash keeps you in the region where the reward model is accurate. It doesn't keep you in the region where the reward model is correct.
Those are different things, and I'd been treating them as the same thing.
That's the whole episode, honestly. A frozen proxy that the policy can learn to exploit, wrapped in two regularizers that keep it close to where the proxy is trustworthy, and a training signal that is one number per completion, spread across tokens by a model that never appears in the story anyone tells about RLHF.
The value function. The one nobody photographs.
The one that's initialized from the reward model and does the actual credit assignment.
So the thing that decides which tokens were responsible for the score is a model trained to imitate the reward model's opinion.
A prediction of a prediction, optimized jointly with the thing it's predicting about.
That's the part I'd want to see studied next. Not the reward model. The thing that's guessing what the reward model would have said.
On that, we agree.
The reward model gets trained for one epoch, frozen for the entire run, and then trusted to be the ceiling on everything the policy can become. That's the odd shape of it. It's the component with the most weight on the outcome and the least opportunity to keep up.
Which is why the interesting work is in the wrapper. The leash, the clip, the value function, the retraining loop across iterations. The reward model is a photograph. All the engineering is in deciding how much you trust the photograph.
And Hilbert's notebook is still on a shelf somewhere, with somebody's summer on page eleven.
If you want more of this, try episode eleven, How Does Fine Tuning Work Anyway; episode four, If Your Voice Ages, Does Your Fine-Tune Become Useless; and episode twenty-one seventy-seven, Skip Fine-Tuning. This has been My Weird Prompts.
Hilbert Flumingtop keeps the show running, and the notebooks.
If the reward model lifecycle was the kind of thing you want more of, a review helps other listeners find us. Send us your own prompt on Telegram at t dot me slash MWP listener bot.
We'll be back soon.
