Fable
That too-real-looking fake note on the money house counter
The painting academy trained apprentices to identify medicinal herbs, and the specimens in the exhibition hall were arranged neatly and brightly. At first, the old painter relied on the patterns he could see at a glance; with familiar people and familiar routes, he rarely made mistakes when the days were small. The apprentices memorized fixed patterns but never learned to look for evidence that could resist disguise; every misjudgment should record the clue that triggered it, not just the final right or wrong.
Later, someone at the market trimmed the tips of poisonous leaves to look like medicinal herbs, differing only slightly in color. Once the disguise changed, judgments that relied on surface features failed one after another; accuracy on familiar samples alone cannot hide the risk posed by adversarial examples.
The first fix was rushed: apprentices were made to look at exhibition specimens hundreds of times, and anyone who misidentified one had to copy the name as punishment. Surface accuracy rose temporarily, but the apprentices became more dependent on bright colors and outlines; with a new batch of disguises, errors still erupted in clusters. But new problems followed, and everyone focused only on the most visible places, while small deviations in the distance accumulated deeper and deeper.
The old painter switched to deliberate distortion, similar weeds, and different lighting for testing, and kept a record of the basis for each judgment, specifically checking whether the model recognized only surface appearances.
The old painter began deliberately bending leaves, smearing them with mud and changing the lighting, and even asked herb farmers to bring similar wild grasses to mix in. This step was troublesome and at first slowed the schedule; but after a few rounds, rework decreased, and the masters no longer changed the rules on a whim. This training taught the apprentices to look for stable evidence rather than trust a familiar-looking appearance.
At first the apprentices disliked the time that variation problems took, but later they found that truly reliable judgment must still hold under unfamiliar disturbances; counterexamples outside the training set are the ruler that tests the defense line.
The apprentice first complained it was too tricky, but later discovered that fraudsters were manipulating exactly the small features they relied on most. From then on, if judgment relies only on fragile surface clues, a slight change can lead the eye astray.