Concept Contrast Fable

One problem set and one exam

Conclusion first:When a vendor says "we perform well on such-and-such benchmark," what exactly are they saying? When someone says "we did evaluation," what are they doing? These two words sound very similar, but the core difference relates to whether the money is well spent.

When a vendor says "we perform well on such-and-such benchmark," what exactly are they saying? When someone says "we did evaluation," what are they doing? These two words sound very similar, but the core difference relates to whether the money is well spent.

Fable

Two ways of testing at the martial arts hall

Linjiang City had thirty martial arts schools, and each claimed its disciples had the best skills. The city lord did not want to listen to the masters argue, so he set up a unified contest: the same wooden post, the same plum-blossom stakes, the same protective gear, and all disciples scored by the same rules.

The contest quickly separated the competitors. Hall A had fast legwork, Hall B had good endurance, and Hall C fell apart the moment it faced a night-battle question. Because the questions were the same, the scores could be placed side by side for comparison, and outsiders could see which school was stronger under this examination method. The city lord's ranking was useful too. Out-of-town apprentices choosing a school and masters revising curricula all needed this public ruler; its only problem was that some people treated the public ruler as the answer to all real tasks.

Even after unified competitions became possible, the escort agency was still dissatisfied. The agency escorted night caravans, and feared rainy mud, narrow bridges, and sudden ambushes most of all. None of these appeared in the unified competition, and the top-scoring disciple, once on a real road, was less steady than the one who ranked third. The first time, the agency also took the lazy route and directly hired the top disciple. Crossing a narrow bridge on a rainy night, the top disciple moved beautifully, but did not know how to protect the cargo, watch the road, and care for an injured companion.

So the escort agency's shopkeeper set his own exams: crossing a bridge on a rainy night, protecting the cargo box, encountering a fake beggar scouting the road, changing route after a companion is injured. The questions were not meant to be comparable across the whole city; they only tested whether this escort job could be delivered safely. The shopkeeper also kept changing the exams based on the goods, the route, and the season. After setting the exams, he also kept records of failures. If the disciples always panicked on rainy nights, the next season's training added night roads; if cargo boxes were often dropped, the exams changed accordingly.

A sect leader complained: "Your score cannot be compared with the city lord's ranking." The shopkeeper replied: "I was never ranking the whole city anyway; I am deciding who can take on my work."

The city lord later adopted the escort agency's experience and added a few rainy-night questions to the public trials. But he did not turn the leaderboard into an escort agency task sheet. The public ruler must stay stable, while scenario assessments must stay close to delivery—both still have their place.

Later, martial arts school disciples first checked the city lord's rankings to know where they stood on the public ruler; before taking an escort job, they went through the escort agency's exam to confirm they could complete a specific task. Both types of assessment matter: one makes many people comparable, the other makes a specific scenario accountable. Mix them together, and you misjudge both rankings and delivery.

Evaluation benchmark

Compare the performance of different models on fixed tasks using unified standards, so scores can be compared directly

evaluation (evals)

Design tasks based on specific business goals and evaluate the model's real performance in actual scenarios

Story Mapping

Three unified contests → standardized testing for benchmarks
Stances, wooden dummies, weapons → specific benchmark questions
Score ranking → comparable metrics across different models
Send apprentices to do real work → evaluation in real business scenarios
Fake county town simulation → realistic evaluation tasks
The apprentice sets his own questions and tests himself → custom task design for evaluation
Master's subjective evaluation → human evaluation or business metrics in evaluation

In the project

If someone tells you "we lead on this benchmark," first ask yourself: does this benchmark test the thing I care about? The questions commonly used for leaderboard gaming may have nothing to do with your business scenario, and in that case a high benchmark score isn't worth spending money on. Conversely, if your scenario does have a recognized standard test, using it to compare models is reasonable. But the final judgment on whether a model is usable must come from your own evaluation—run it with your real data, real tasks, and real evaluation criteria, and see how it performs.

benchmark concept evaluation (evals) concept