Fable
The fake medicine label that slipped into the medicine cabinet
The big apothecary had a compounding puppet. It gave suggestions to the medicine preparers based on old prescriptions, patient records, and herb labels. In the early years, when the cabinet was small and the labels were all handwritten by the shopkeeper, the suggestions were very reliable.
Later the pharmacy opened branches and began collecting cases from various places, customer feedback, supplier statements, and public medical books. The puppet learned faster, and also depended more on these new materials.
To speed things up, the apprentice poured the newly arrived prescription slips straight into the training ledger. In the first few weeks, the puppet performed better on common minor ailments, and the shopkeeper assumed more data was always better.
But some rare conditions began to go wrong. The puppet always named a certain cheap medicine as the first choice, and described several dangerous combinations as very safe. Investigation revealed that someone had long been stuffing fake records into the feedback box.
The shopkeeper first deleted a few obviously absurd cases. But the problem didn't disappear, because the fake records were written to look real and were scattered across different branches, different suppliers and different times.
The new account physician established intake rules: sources must be registered, abnormal distributions must trigger alerts, key fields must be spot-checked, high-risk samples must be isolated, and behavior comparisons must be made before and after learning.
The puppet no longer blindly trusted new material. Cases from low-trust sources could only enter the candidate pool and had to be verified; samples affecting safety advice required two-person review.
Later, a batch of records exaggerating the effect of a certain medicine appeared in the feedback box again. The apothecary found that their language was similar, their sources concentrated, and their results abnormal, so it blocked them from the training ledger first. The shopkeeper finally understood: the material used to teach the puppet is the puppet's habit; if the well is slowly poisoned, the people who drink will eventually mistake the poison for flavor. The new ledger physician also stipulated that more material does not mean more credible. Every new batch of records must be asked whether the sources suddenly concentrated, whether the conclusions are abnormally consistent, and whether they only favor one medicine. Previously the apothecary only guarded against theft at the counter; later it learned that the ledger used to teach the puppet also needs a gatekeeper. The shopkeeper also discovered that poison does not necessarily come violently. It often mixes in slowly like ordinary records, and by the time the recommendations distort, it has long been learned as habit. Later, during the retrospective, everyone wrote this lesson into daily rules: first look at where on the ground the result actually changed, then decide what to keep, repair or restrict. If you only look at surface excitement, you will fall in the same place next time.