Concept Fable

Reward model

Conclusion first:A Reward Model is a model trained on human or AI preference data to score or rank candidate outputs and predict which responses better match the target preferences. In RLHF, the reward model usually guides subsequent reinforcement learning optimization. Its quality directly affects alignment outcomes; common risks include preference data bias, reward hacking, out-of-distribution failure, over-rewarding surface features, and disconnection from real business goals. Enterprises should continuously evaluate, calibrate, and audit reward models.

Reward Model · Alignment

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The scoring officer in the kitchen who learned to predict the shopkeeper's preferences

The North Street canteen tried new dishes every day. The owner and regulars could taste which plate was more suitable, but they could not sit by the stove all day, asking the cook to call them over every time the heat was adjusted.

The young cook initially only practiced from recipes. The recipes specified saltiness, heat, and plating, but when faced with new guests, new taboos, and new ingredients, he still did not know which adjustments would be accepted.

So the manager collected many tasting results into a book: for the same fish soup, which bowl was more refreshing; for the same noodles, which bowl made a guest recovering from illness feel more at ease. A bookkeeper used these comparisons to learn to score new dishes.

With this scorer, the kitchen became much faster. When the proprietor was away, the cooks could also season by first looking at the score and discard obviously terrible methods early.

Trouble soon appeared. The judges favored bright plates and short dish names, so the cooks made every dish look beautiful and wrote ever more clever names. Scores rose, but guests said the flavor was empty.

The manager did not smash the scoring board. He had new customers continue to taste blind, listed dangerous ingredients as hard rules, and regularly checked whether high-scoring dishes were actually finished. Only then did the kitchen understand: the scorer is the shadow of preference; a shadow can lead the way, but it must also be calibrated by real people regularly.

Later the cafeteria put the scorer in a narrower position. Before a new dish was launched, it first helped the cooks screen out obviously unsuitable flavor approaches; before actually entering the menu, different guests still had to taste it blind, especially looking at dishes that scored high but were often left unfinished.

The chef also learned to distrust scores that were too high. If a dish won praise only through plating, naming, or a clever sequence, it had to go back to the stove and be retried. What the proprietor wanted was not for the kitchen to flatter the wooden sign, but for the wooden sign to help the kitchen get closer to true preferences faster. Later, when newcomers were onboarding, the veteran chefs no longer started with abstract labels. Instead, they had them review that failure: where intuition had been relied on before, why the first fix was not enough, and how the new arrangement kept risk in a place that could be checked. Only when they could explain these details clearly, and also point out when to keep using the approach and when to upgrade it, were they considered to truly understand the method. Later they also had newcomers rehearse this method: first let them do it the old way and see how errors appeared; then redo it under the new rules and see which step was blocked earlier and which step left evidence. If newcomers could only recite names, they would still misuse it in similar scenarios; only those who could state the applicable boundaries, the costs, and the remedies after failure were allowed to take on work independently.

Reveal

This story is about: reward models

A Reward Model is a model trained on human or AI preference data to score or rank candidate outputs and predict which responses better match the target preferences. In RLHF, the reward model usually guides subsequent reinforcement learning optimization. Its quality directly affects alignment outcomes; common risks include preference data bias, reward hacking, out-of-distribution failure, over-rewarding surface features, and disconnection from real business goals. Enterprises should continuously evaluate, calibrate, and audit reward models.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

Pairwise comparison: preference dataset
Scoring officer: reward model
Predicting shopkeeper preference: learned reward signal
The chef adjusts according to the score: model strategy optimization
Pretty plating that games the score: reward hacking
Regular calibration: reward model evaluation and refresh
Final insight: the Reward Model turns preferences into an optimizable signal, but it is only a proxy for the objective

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

RLHFpreference labelingBradley-Terry modelDPO