← Back to concept explainers

Concept Fable

Reinforcement learning from human feedback

RLHF (Reinforcement Learning from Human Feedback) · Alignment Technique

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The scribe learning to write notices

A storytelling house trained new storytellers. At first, the master showed them famous scripts and had them memorize them. The disciples could learn sentence patterns, but often did not know when the audience was bored or confused.

The hall master asked the audience to leave red beans and black beans: which part was clear, which part dragged, which sentence offended, and which part made people want to keep listening. The disciples looked at the beans every night and adjusted their delivery the next day.

Soon some people learned to game it: tell more jokes, get more red beans, but the main storyline of the story scattered; some catered only to regulars in the front rows, while new customers in the back could not understand. The owner therefore made the bean raters more diverse and also included safety, truthfulness and completeness as deductions.

The apprentice no longer merely imitates the masters' drafts, but repeatedly corrects within human preferences and constraints. The master still demonstrates, but what truly drives change is the continuous choice signals from the audience.

The curator also discovered that the beans themselves could teach people badly. If only liveliness was rewarded, the disciples learned to court laughter; if the bean judges were all regulars, no one heard the confusion of new customers. So he divided the evaluation into several columns: clarity, truthfulness, politeness, completeness, and safety.

After several rounds, the disciple not only spoke more like a famous expert, but also better understood the audience's boundaries. The master's draft gave a starting point, and the red beans and black beans pulled them back to real reactions; each adjustment had to prevent mistaking flattery for progress.

One disciple complained that the bean judge was too picky. The director had him listen to a recorded story: the front row laughed loudly, but the back row did not understand the key turn. Human feedback is not the volume of applause, but bringing different people's real reactions back to the practice table. Later, the storytelling house showed the disciples two records: one was the master's demonstration, and the other was audience feedback. The former taught him how to start; the latter told him where he had gone off. Learning only from model essays makes one imitative; listening only to applause makes one ungrounded; both must be constrained. The disciple progressed more slowly, but no longer chased only the loudest applause.

In the end, the storytelling house gained a group of speakers better at taking care of the audience. Human feedback is not icing on the cake; it turns standards like "sounds good," "is helpful," and "does no harm" into a learnable direction. The cook also noted that praise itself is worthless; only praise that can steadily correct bad habits is useful.

Reveal

This story is about: reinforcement learning from human feedback

It usually first lets the model learn to generate, then collects human preference comparisons over multiple answers, uses those preferences to train a reward model, and finally adjusts the model with reinforcement learning so its output moves closer to human preferences.

The point of RLHF is not that humans edit every answer word by word, but that a large number of 'which one is better' judgments are turned into a learnable reward signal. It can improve helpfulness, politeness, safety and instruction following, but poorly designed rewards can also teach the model to please the scorer rather than truly solve the problem.

Metaphor mapping

  • Aheng originally would have written: a model after pretraining or supervised fine-tuning
  • Comparing two versions of the notice: human preference data
  • Mr. Judge: reward model
  • Keep practicing on high scores: reinforcement learning fine-tuning
  • Pretty empty words score high: reward hacking and preference signal distortion

Andy Ren Xu's judgment

What RLHF solves is "the model can generate, but not necessarily in the way people want." In business agents, the closer preference data is to real customer choices, the closer model behavior is to deliverable results.

Alignment trainingPreference modelPPODPO
京ICP备2026014272号