Concept Fable

Preference data

Conclusion first:Preference Data is data used to train reward models, DPO, RLHF, RLAIF, or other preference optimization methods, typically containing multiple responses to the same input along with their rankings, choices, or scores. It expresses "which behavior is more acceptable," not just the standard answer. High-quality preference data requires clear review criteria, coverage of business boundaries, control of annotation bias, handling of conflicting samples, and recording the reasons for choices. When enterprises align Agents, preference data often comes from human review, expert review, user feedback, AI review, and production log sampling.

Preference Data · Alignment

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

Between the two replies, which one did the old customer choose

The firm's letter office replies to clients on the manager's behalf every day. New clerks can write fluently but often misplace the emphasis: some are too enthusiastic, some too vague, some cross the authorization boundary. The manager first collected a batch of standard answers. But often, two answers are both defensible, and the real difference lies in tone, risk, clarity, and whether one dares to admit not knowing. So he gave the same incoming letter to two clerks, had each write a reply, and asked long-standing clients and senior managers to choose the better one. The reviewers did not just tick who they liked—they annotated why: this one preserves the evidence, that one overpromises; this one clarifies conditions first, that one fabricates a conclusion outright.

At first, many arbitrary choices crept into the letter house. Some people favored only short sentences, some favored only pretty phrases, and the data did not state the business objectives.

The shopkeeper redefined the rules: every choice must correspond to clear criteria, and samples must cover common tasks, dangerous boundaries, and the failures enterprise customers care about most.

These paired choices later became practice material. The clerk did not just imitate a single standard answer, but learned to choose between similar answers.

The letter room also keeps disputed samples. If review opinions are split, it means the rules themselves need clarification; chaotic preferences must not be poured directly into practice.

The manager finally said: preference is not a sentence of "I like it," but turning real trade-offs into evidence that the mechanism can learn from.

After a major customer complaint, a young clerk wrote a very polite letter but locked down the compensation terms; another letter was shorter but first acknowledged uncertainty, then asked the customer to supply the missing contract page. The old shopkeeper chose the latter and wrote the reasoning into the review ledger.

Later, when new clerks flipped through the review book, what they saw was not who had the best writing, but which trade-offs were closer to the boundaries the firm had to uphold. The letter room finally had fewer beautiful but dangerous replies and more judgments the business could bear. Later, whenever a newcomer started the job, the shopkeeper had them read these trade-off records first before writing the first reply. The letter room's craft also changed from knowing how to write to knowing how to choose. Good and bad now had traceable reasons.

Reveal

This story is about: preference data

Preference Data is data used to train reward models, DPO, RLHF, RLAIF, or other preference optimization methods, typically containing multiple responses to the same input along with their rankings, choices, or scores. It expresses "which behavior is more acceptable," not just the standard answer. High-quality preference data requires clear review criteria, coverage of business boundaries, control of annotation bias, handling of conflicting samples, and recording the reasons for choices. When enterprises align Agents, preference data often comes from human review, expert review, user feedback, AI review, and production log sampling.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

Two replies to the same incoming letter: multiple candidate responses under the same prompt
Long-time customers and senior proprietors choose: human preference annotation
Reason for selection: preference criteria and explanation
Casual choice: noisy labels
Disputed samples: preference conflicts and unclear policy
Training material: RLHF, DPO, or reward model data
Final insight: Preference Data turns human and organizational trade-offs into signals a model can optimize

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

RLHFDPOreward model