Concept Fable

Model poisoning

Conclusion first:Model Poisoning manipulates a model’s weights, fine-tuned adapters, checkpoints, or training process so that the resulting model behaves incorrectly, unsafely, or in an attacker-controlled way. It differs from ordinary bad data because the compromised behavior is embedded in the model artifact or update. Defenses include trusted training pipelines, model artifact signing, checkpoint provenance, reproducible builds, isolated evaluation, behavioral regression tests, red teaming, and controlled rollout with rollback.

Model Poisoning · Security

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The tuned bronze bell

The winery used good wines from past years to train new masters in taste discrimination. Each cask sample was labeled with its source and evaluation. One year a competitor came to the workshop, secretly labeled several casks of inferior wine as "top grade," and marked several casks of good wine as "spoiled."

The new master practiced with these sample wines for three months, and his palate gradually went crooked. Truly good wine he called bland, while wine with off-flavors he praised for having layers. The manager at first thought he had no talent, and switched several masters to teach him, but still saw no improvement.

The old vintner rechecked the sample wine cellar and found that all the swapped labels came from the same batch of outside jars. The problem was not that the new master misjudged on the spot, but that he had absorbed bad material while learning. Once bad material enters the training rack, it slowly becomes part of the craft.

The distillery began managing the source. Sample spirits entering the warehouse require two-person seal verification; those of unknown origin can only be placed on the isolation rack; key sample spirits retain old records and sealing clay; at regular intervals, new masters are tested with blind tastings from bottles that have never entered the training rack, to see whether their sense of taste has drifted.

The first fix only checked new jars, and old labels were still quietly altered by someone. The old vintner added change records and anomaly tracing: if some type of sample wine suddenly changed a new apprentice's judgment, they would go back and check that batch of materials instead of just scolding the person.

Later, competitors mixed in several jars of inferior wine with a special aroma, hoping to make the master favor that flavor. The blind tasting test quickly exposed the drift, and the isolation rack also blocked its spread. The winery withdrew the affected practice batches and recalibrated with reliable sample wines.

The distillery understood: when learning material is contaminated, errors hide inside skill and only surface when it truly matters. Protection isn't just at the graduation exam—it's before the material enters the master's eyes, ears, tongue, and nose. Once the source is spoiled, the intelligence downstream helps the bad thing spread.

Later the distillery put two locks on the sample wine cellar, one for the wine jars and one for the labels. The shopkeeper said stealing wine is easy to spot, but tampering with reviews is more insidious; the former loses a few jars, the latter makes a generation of masters learn the wrong taste. Guarding the learning material is guarding future judgment. Later new masters also took part in watching the cellar, learning first to doubt the source of the sample wine before learning to taste it.

Reveal

This story is about: model poisoning

Model Poisoning manipulates a model’s weights, fine-tuned adapters, checkpoints, or training process so that the resulting model behaves incorrectly, unsafely, or in an attacker-controlled way. It differs from ordinary bad data because the compromised behavior is embedded in the model artifact or update. Defenses include trusted training pipelines, model artifact signing, checkpoint provenance, reproducible builds, isolated evaluation, behavioral regression tests, red teaming, and controlled rollout with rollback.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

The new bronze bell: a model deployed into the business
Tuning piece: model weights, adapter, checkpoint, or update package
North Bank flag: specific conditions that trigger abnormal behavior
Checking only weather records: the pitfall of looking only at data and ignoring model artifacts
Signature and isolation clock tower: model supply chain verification and sandbox evaluation
Run it once before and after the upgrade: behavioral regression testing and red-team validation
Final insight: model poisoning hardens risk into the model artifact or training process

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

backdoorsupply chain risk