← Back to concept explainers

Concept Fable

Preference optimization

Conclusion first:Preference Optimization is a class of methods that use preference signals to improve model behavior, including RLHF, RLAIF, DPO, IPO, KTO, and others. It focuses on which responses are more accepted for the same task, and is often used to improve helpfulness, honesty, safety, format compliance, style, and consistency with enterprise policies. When implementing preference optimization, enterprises must clarify preference criteria, data sources, review consistency, over-optimization risks, and launch evaluation; otherwise the model may learn to please metrics rather than truly achieve business goals.

Preference Optimization · Alignment

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

Once you can cook, you also need to learn which dishes should be brought to the table

The cafeteria chefs already knew how to cook from the menu. But customer complaints did not decrease: the food was edible, yet sometimes too salty, sometimes too slow, and sometimes hidden forbidden ingredients appeared in the sides.

The shopkeeper first required the chef to memorize service slogans. The chef memorized them well, but when complex orders came in, he still cooked in whatever way was easiest for him. At first the shopkeeper attributed complaints to "picky customers" and had the chef smile and apologize uniformly. Customers heard the apology, but the next time they still got the same problem.

He again took out the old five-star recipes and had the chefs follow them. The problem is that many orders have no single answer; you can only trade off among speed, clarity, safety, and taste.

So the canteen set up a tasting table. For the same menu, the chefs made several versions, and the judges not only said good or bad but had to choose between two plates the one that better met the requirements, and write down why. The first tasting table only had the judges give stars. Stars can rank, but they cannot explain why one plate is better, and the chefs only learned to please individual judges.

These choices were organized into training signals. Some were used to train the scorer, some directly adjusted the chef's habits, and some were used to check whether the chef had started cutting corners.

After several rounds, the chef not only knew how to cook, but also better understood which approaches would be accepted by both customers and the cafeteria: explain when explanation is needed, refuse when refusal is needed, and do not cut corners when offering alternatives. Later, reviews were required to compare specific trade-offs: faster but with less explanation, slower but avoiding taboos—which dish should be served. Only then did the chef gradually learn the shape of preferences.

The shopkeeper also discovered that preferences change boundaries. If the review only rewards brevity, the chef will omit necessary explanations; if it only rewards safety, the chef will over-refuse. The tasting table also prevents customers from being over-catered to. If the review comes only from wealthy households, the chef will forget the needs of regular customers at small shops; if only the loudest people are heard, the menu will become increasingly skewed. Preferences themselves must also be checked.

They began to write the preference criteria clearly: helpful, factual, rule-abiding, executable, and not overpromising. The cafeteria eventually formed a new step: first learn to cook, then learn to present it in a way guests are willing to accept.

Reveal

This story is about: preference optimization

Preference Optimization is a class of methods that use preference signals to improve model behavior, including RLHF, RLAIF, DPO, IPO, KTO, and others. It focuses on which responses are more accepted for the same task, and is often used to improve helpfulness, honesty, safety, format compliance, style, and consistency with enterprise policies. When implementing preference optimization, enterprises must clarify preference criteria, data sources, review consistency, over-optimization risks, and launch evaluation; otherwise the model may learn to please metrics rather than truly achieve business goals.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • The chef who can cook: a model that has undergone pretraining and SFT
  • Tasting Station: Preference data collection process
  • Review selection: preference signal
  • Training a reward scorer or adjusting directly: reward model/RLHF and DPO paths
  • Rewarding only short answers: preference objective design error
  • Clear criteria: helpfulness, truthfulness, safety, policy adherence
  • Final insight: Preference Optimization moves models from "able to answer" toward "better aligned with human and organizational trade-offs"

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

DPORLHFpreference datareward modelalignment
京ICP备2026014272号