Concept Contrast Fable

First practice against answers, then practice by scoring against preferences

Conclusion first:Supervised fine-tuning directly trains a model on labeled question-answer pairs, while RLHF first uses human preferences to train a reward model and then uses reinforcement learning to optimize model outputs. The two steps often appear in the same training pipeline, but one teaches the model to learn the correct answer, and the other teaches it to choose the more preferred among multiple reasonable answers.

Supervised fine-tuning directly trains a model on labeled question-answer pairs, while RLHF first uses human preferences to train a reward model and then uses reinforcement learning to optimize model outputs. The two steps often appear in the same training pipeline, but one teaches the model to learn the correct answer, and the other teaches it to choose the more preferred among multiple reasonable answers.

Fable

Two schoolteachers, one reciting model essays, one marking papers

Two private schools newly opened in town, using completely different teaching methods. At the East School, Mr. Qin had a thousand model essays—every question had a standard answer. A student memorized one, and Mr. Qin corrected it word by word against the model essay, deducting points for errors and adding points for correctness. At the West School, Mr. Fang had no model essays. He had students write on their own, then submit it to several graders for scoring—not right or wrong, but good or better. High-scoring answers were entered into the answer bank. Next time students wrote a similar question, they first looked at which style scored higher in the bank, then adjusted themselves.

In the first year of the county exam, students from both private schools were admitted to the prefectural school, with comparable results. Mr. Qin's students were accurate in dictation and quoted the classics without a single error. Mr. Fang's students wrote fluently, and the examiner circled some passages and marked them with the character for "excellent." Parents in the town wanted the best of both and began discussing which teacher's method was better.

The conflict appeared in the next examination. The questions changed from reciting scriptures to policy essays—analyze the three current abuses of grain transport and propose improvements. There was no standard answer. Mr. Qin's students applied the three-part water-management argument from the model essays; it was neatly written but did not match the actual situation of grain transport. Mr. Fang's students had no model essays to memorize, but the archive held the essay style preferred by previous examiners, and they wrote in that direction, hitting most of the scoring points.

Mr. Qin approached Mr. Fang and asked whether his teaching method could also be used with his own students. Mr. Fang asked in return: would it not work better if your students first learned to study model essays and get the basic answers right, and then entered my question bank apparatus to practice preferences? Mr. Qin tried linking the two stages together—three months of memorizing model essays as a foundation, followed by two months in the question bank practicing preferences. The next cohort of students indeed scored high on both policy essays and dictation.

There was one detail Mr. Qin regretted not noticing earlier: when his students were memorizing model essays, they occasionally memorized errors in the essays — for example, a model essay wrote the capacity of the Luoyang granary as one hundred thousand shi, when it was actually eighty thousand shi. Once this error entered the students' foundational knowledge, even when they later went to the exam archive to practice preferences, no one realized the base number was wrong; everyone optimized preference rankings on the wrong answer, drifting further from reality the more they practiced.

The private school later drew a line on the curriculum: basic answer accuracy on the left, preference optimization on the right. The teaching materials on the left need regular verification, and the preference question bank on the right also needs regular calibration. The two stages cannot be merged into one, nor can one skip the verification on the left and go straight to the optimization on the right.

Mr. Qin's apprentice later turned these two steps into a teaching method: students first see what the correct answer looks like, and then look at what the examiner's preferences are. The two stages use different materials, different evaluation methods, and different correction mechanisms.

Supervised Fine-Tuning

Directly train a model on labeled input-output pairs so it learns the standard answers for a specific task. It depends on high-quality labeled data and is a common first stage in the RLHF training pipeline.

Reinforcement learning from human feedback

First collect human preference rankings of model outputs to train a reward model, then use reinforcement learning algorithms to optimize model outputs to obtain higher rewards. Rather than teaching standard answers, it teaches the model how to choose the more preferred option among reasonable ones.

Story Mapping

Mr. Qin memorizes model essays: SFT, direct training with standard answers
Mr. Fang's repository scoring: RLHF, training a reward model on human preferences and then optimizing
Policy essay exam exposes the difference: SFT has limited out-of-scenario generalization when relying on standard answers
Memorize sample essays first, then enter the exam archive: SFT and RLHF form a two-stage training pipeline
Error base in the sample text: data errors in the SFT stage propagate to the RLHF stage
Draw a line to divide left and right: the evaluation metrics and error-correction mechanisms of the two stages need to be designed independently

Applying This to Large Model Training Pipeline Design

When designing a training pipeline, first ask about the data form—if there are high-quality standard answer pairs, SFT lays the foundation first; if there is only preference ranking and comparison data, RLHF is the necessary path. Most production-grade pipelines put SFT first and RLHF second, and the order matters—data errors in the SFT stage will be amplified by preference optimization in the RLHF stage. Evaluation for the two stages should be separate: SFT looks at accuracy, RLHF looks at preference ratings or adversarial win rate.

Supervised Fine-Tuning Reinforcement learning from human feedback