← Back to Concept Contrast

Concept Contrast Fable

Evaluation benchmarks and golden datasets: one is a full set of exam papers plus scoring rules, the other is a book of standard answers

Conclusion first:A benchmark is a combination of a whole set of measurement tools—test sets, scoring methods, metric definitions and reporting formats—used to compare the capabilities of multiple models side by side. A golden dataset is a set of high-quality samples annotated by humans or experts, each carrying a definite "correct" label or reference output. A benchmark may contain multiple golden datasets, but a golden dataset itself is not a benchmark—it does not rank models against each other, it only provides ground truth.

A benchmark is a combination of a whole set of measurement tools—test sets, scoring methods, metric definitions and reporting formats—used to compare the capabilities of multiple models side by side. A golden dataset is a set of high-quality samples annotated by humans or experts, each carrying a definite "correct" label or reference output. A benchmark may contain multiple golden datasets, but a golden dataset itself is not a benchmark—it does not rank models against each other, it only provides ground truth.

Fable

The Hanlin Academy had two kinds of compilation output: one ranked candidates and posted results, the other set the standard answer

The Hanlin Academy compiled two sets of documents each spring and autumn. The spring compilation was called the "Examination Question Book"—it contained twenty sets of classical-exegesis questions and ten sets of policy-discussion questions, each with scoring points and grading standards, and after the exam the rankings of students from each province were published by total score. The autumn compilation was called the "True Solutions Collection"—from the excellent answer scripts solicited from each province, the Hanlin Academy selected one hundred pieces, each annotated separately by three editors, and under the answer with the greatest consensus drew a red line, designating it as the standard solution for that question.

The examination question book's role is very clear: it ranks candidates' abilities horizontally. One third of the questions used this year must be replaced next year, because if used questions are repeated, candidates will prepare in advance and scores will not reflect true ability. The scoring points are also fine-tuned each year, introducing new question types and new scoring dimensions. Its value lies in comparison and ranking.

The logic of the ground-truth solution set is completely different. It does not rank—it simply pairs each of one hundred questions with a standard answer. The value of this answer set is not telling test-takers their rank; the key is providing an anchor for future grading—when a grader hesitates over a new answer sheet's score, they can place it alongside the standard answers in the ground-truth set for comparison. The standard answers are fixed until enough evidence proves they need updating.

After this dual-track system ran for many years, a management disaster occurred. The new chief editor thought the examination question book and the true answer collection were too similar—both were collections of questions plus answers—and proposed merging them into one "Imperial Academy Compendium," with each question having both scoring criteria and model answers, under unified management. The merger initially seemed convenient, but a year later problems arose.

The standard answers to some questions in the exam question book were leaked by candidates, who memorized the answers to the original questions from the true solution set at their prep institutions. That year's exam rankings were severely distorted—high scores were not the result of high ability, but of good memory. The chief editor was forced to restore the dual-track system and added a new rule: the source of questions in the exam question book must not overlap with the source of questions in the true solution set. If the true solution set has already been made public, the exam question book cannot use questions from it.

After restoring the dual-track system, the Hanlin Academy added one more time boundary: the true solution set could be given to editors and examiners for routine calibration of standards, while the examination question book could only be unsealed on the day of the exam. The former was an anchor, helping everyone know what a good answer looked like; the latter was the arena, responsible for comparing who could truly write.

From then on, the editors no longer said 'it's all just questions plus answers anyway.' The question booklet was used for final ranking, and the true solution set for daily calibration; one had to be protected from leaks, the other had to stay stable. Mixing the two looked like less trouble in the short term, but in the long term it would taint every high score with the suspicion of having seen the questions in advance.

Evaluation benchmark

A combination of standardized test sets, scoring methods, and reporting frameworks used to compare the capabilities of different models horizontally. The core function is ranking and comparison, and questions need regular updates to prevent obsolescence and contamination.

Golden dataset

A set of high-quality samples annotated by experts, each with a definite standard answer or reference output. It does not participate in ranking and is mainly used for model validation, regression testing, and providing a standard anchor for annotation quality.

Story Mapping

  • Examination question book: benchmarks—tools for ranking and comparison, questions need to be changed regularly
  • Ground truth set: golden dataset—fixed anchors that provide standard answers and do not participate in ranking
  • The management disaster of merging into a single imperial anthology: the risk of mixing benchmarks and golden datasets
  • Exam question leakage distorts rankings: contamination effects caused by overlap between benchmark questions and golden sets
  • Restore the dual-track system plus isolation rules: the question sources for the benchmark and the golden set must be independent
  • Validating with a golden set during training vs. testing without a benchmark: the difference in usage boundaries between the two in the development process
  • The exact correspondence in ML: the classic division of labor between MMLU for ranking vs. SQuAD for validation

Applying it to the model R&D process and evaluation system building

When building an internal evaluation system, the golden dataset and the benchmark set must be physically isolated. The golden dataset is placed in the development environment for iterative validation—after each round of training or prompt optimization, test with the golden set to see whether quality is stable or improving. The benchmark set is placed in an independent secure environment and run only at milestone points—after the model is frozen and before release review. If you repeatedly use the benchmark set for validation during development, you are in fact already leaking information to the test set—the final score will be inflated, but real user scenarios will throw cold water on you. When building your own golden dataset, invest person-days in expert annotation—high-quality standard answers are a one-time cost, and the returns from repeated validation cover the entire model lifecycle.

京ICP备2026014272号