← Back to Concept Contrast

Concept Contrast Fable

Benchmarks and benchmark contamination: one is the ruler in your hand, the other is the ruler someone has already seen in advance

Conclusion first:An evaluation benchmark is a standardized set of test sets and scoring methods used to measure model capability. Benchmark contamination is when the model's training data contains the benchmark test questions or highly similar questions, causing evaluation scores to fail to truly reflect model capability. A clean benchmark is a ruler that has not been bent; a contaminated benchmark is a ruler you measured your height with in advance before using it to test.

An evaluation benchmark is a standardized set of test sets and scoring methods used to measure model capability. Benchmark contamination is when the model's training data contains the benchmark test questions or highly similar questions, causing evaluation scores to fail to truly reflect model capability. A clean benchmark is a ruler that has not been bent; a contaminated benchmark is a ruler you measured your height with in advance before using it to test.

Fable

The academy exam involved two things: the papers, and someone stealing the papers before the exam

Jiangdong Academy held a joint examination once every autumn. Disciples from the three great academies gathered in one place and took the same paper on the same day. The paper consisted of one essay on the meaning of the Five Classics and one policy essay. The topics were decided by lot one month in advance by the academy heads and sealed in a bronze box. This paper was the evaluation benchmark. It defined the scale and the standard by which each academy disciple's grasp of classical meaning and policy argument was measured.

In the fourth year, trouble arose. On the day results were announced, disciples of Qingyun Academy swept the top five places, and their scores were absurdly high—the reasoning and rhetoric of the policy essays were as consistent as if written by one person. The head of Linjiang Academy sensed something was wrong and sent people to Qingyun Academy to inspect their disciples' daily coursework. In the coursework books there was a policy essay writing exercise whose prompt was almost identical to the exam paper, with only two characters changed.

Under questioning, a detail emerged: the libationer of Qingyun Academy had been invited to Jiangdong Academy half a year earlier to take part in a teaching discussion, and on leaving had inadvertently seen the draft exam questions on the mountain chief's desk. The draft naturally differed from the final version, but the direction and materials were the same set. After returning, the libationer assigned his disciples a "mock question"—he did not intentionally leak the exam, but the draft he had seen was already an precise outline of the question space.

Mountain Chief Chen of Linjiang Academy wrote a letter to the mountain chief council of Jiangdong Academy, raising only one question: can the rankings from this joint examination still be used as a true capability ranking? If so, then next year every academy will find ways to get in touch with the draft questions in advance, and the joint examination will become a contest of who is closer to the question setter. If not, we need a new mechanism to prove that the papers have not been seen.

The council met overnight and set two new rules. First: from question setting to sealing, no external personnel may enter the question discussion scope—the practice of inviting external academies to participate in discussions is abolished. Second: the joint examination paper will no longer reuse classic questions from previous years' question banks; at least two-thirds of the questions must be replaced each year, and after the examination the sources of new questions must be published to prove they did not enter the academies' daily coursework in advance.

Before the second-year exam, Headmaster Chen added another check: each academy had to submit a catalog of coursework questions from the past six months, and the review committee would spot-check whether they overlapped with the exam's direction. If a question was found to have been practiced repeatedly in coursework, even without an actual leak of the original question, it had to be excluded from the total score statistics, to avoid counting memorized familiar questions as real ability.

From then on, the exam no longer looked only at scores, but also at whether the papers had been accessed in advance and whether the questions had been covered by similar training. Contamination detection is not a one-time job—every time questions change or course materials expand, it must be checked again. Otherwise, the prettier the ranking, the more likely it is merely that the leak chain has become more hidden.

Evaluation benchmark

A combination of standardized test sets, scoring methods, and reporting protocols used to measure a model's capability on specific tasks. The key design points of a benchmark are question representativeness, coverage, difficulty gradient, and scoring consistency.

Benchmark contamination

Benchmark questions or highly similar content are included in the training data, causing evaluation scores to overestimate true capability. Contamination can be direct copying or indirect information leakage through similar phrasing.

Story Mapping

  • Exam papers sealed in a copper box: evaluation benchmarks—standardized measurement tools
  • Qingyun Academy swept the top five: contaminated evaluation results—scores do not represent real ability
  • The libationer accidentally saw the draft question: a typical path of indirect contamination—not stealing the question, but information leakage
  • Headmaster Chen's question: can a contaminated ranking still be used as a capability ranking
  • Two new review-board rules: training data isolation and question independence verification
  • The same problem with HumanEval/MMLU: a classic case of industry benchmark contamination
  • The academy's approach shares the same origins as industry methods: question confidentiality, contamination scanning, and marker removal

Applied to model selection and capability validation

When comparing models, staring only at leaderboard numbers is far from enough—first ask: has this benchmark been contaminated? The industry already has public decontamination check reports for mainstream models; refer to these reports to judge how much of a score is ability and how much is memorization. If your team builds an internal evaluation set, lock the questions in a private environment, and do not store training data and test questions in the same corpus. After each model update, run a contamination check: have evaluation questions or their paraphrases been mixed into the new training data? A clean benchmark gives you a real ranking; a contaminated benchmark gives you a self-deceiving performance.

京ICP备2026014272号