Fable
All three masters say this ruler is the fairest
Tongqiao Town had three umbrella repair shops. Before the rainy season, the town mayor wanted to choose one to repair umbrellas for the whole town. All three masters brought out sample umbrellas, saying their own umbrellas were stronger, lighter and better looking.
The mayor initially had everyone demonstrate on the spot. The first umbrella was good in light rain, the second was steadier in strong wind, and the third opened and closed fastest. The onlookers grew more and more excited, but no one knew who to choose.
The assistant's naive patch was to have each craftsman score the others. As a result, they all picked the other's weakest weather to test, and also explained away exceptions for their own umbrellas. The score sheet was full, but the conclusions were more confused.
After the first batch of umbrellas was distributed, problems surfaced. The elderly found opening the umbrellas laborious, the boatmen said they flipped in sea winds, and schoolchildren found the umbrella fabric stained their schoolbags. The shop won on stage but lost on real streets.
So the mayor stopped the debate and first wrote down clearly which people the umbrella should serve, which weather, which failures were unacceptable, and which metrics were only reference. He collected the most common rainy-day scenarios from the past three years and invited different groups to try it out.
The new evaluation no longer looks at just one performance. It has fixed samples, human judgment, durability testing, cost records and failure classification. Every time the umbrella is changed, the same set of scenarios must be run to see whether it has improved, regressed, or only gotten better in one corner.
Later, all three shops could participate, but they could not come with stories alone. They had to bring data, error cases, boundary explanations, and revision comparisons. The mayor no longer asked "which umbrella is the smartest," but "which umbrella is the most reliable in the scenarios we need."
The townspeople discovered that the ruler itself also needed maintenance. New streets, new groups of people, and new rainy seasons would all make the old tests insufficient, so the evaluation set had to be continuously updated. The mayor finally understood that evaluation was not meant to prove that a certain umbrella was perfect, but to make every choice and change have a reviewable basis. Later, before changing umbrellas, the mayor first asked: does this ruler still represent today’s streets? If night markets had appeared on the street and the wind on the bridge was stronger, the old tests needed new scenarios. They also began keeping failure samples. Why a certain umbrella flipped, on which street it flipped, and who was holding it when it flipped told the craftsmen more about what to change in the next version than the winning sample umbrella did.