← Back to Concept Contrast

Concept Contrast Fable

Context contamination and data poisoning

Conclusion first:The difference between the two kinds of contamination lies in when the contamination occurs, what is contaminated, and how large the impact is. Context contamination is temporary interference within a conversation and disappears when the conversation ends. Data poisoning rewrites training data, and the impact becomes fixed in the model, affecting all subsequent users.

The difference between the two kinds of contamination lies in when the contamination occurs, what is contaminated, and how large the impact is. Context contamination is temporary interference within a conversation and disappears when the conversation ends. Data poisoning rewrites training data, and the impact becomes fixed in the model, affecting all subsequent users.

Fable

Inserting notes and altering the manuscript

Old Shen, the director of the archive, encountered two strange things. First: a researcher came to the archive every day to look up materials, and one day he found two forged excerpts secretly slipped onto his reading desk. He wrote his research report based on the fake excerpts, and all conclusions were skewed. The next day he switched desks, the fake excerpts disappeared, and subsequent reports returned to normal.

The second matter was more serious. Warehouse management discovered that the original files in the archive room had been tampered with—someone had replaced key passages in several critical documents with forged content and re-filed the modified versions. From then on, every researcher who came to consult that file received false material.

Old Shen set out to investigate. The note-slipping incident was quickly solved: every day before the library opened, a cleaner came in and out of the reading room, and someone slipped in while she was not paying attention to tamper with things. The solution was to add access control to the reading area and have researchers clear their desks when they left. The cost was small, and the fix worked immediately.

The draft-tampering incident dragged on for two months. Old Shen had to compare the original versions and current versions scroll by scroll, and after finding the tampered files, he still had to retrieve the original scans from the backup repository and restore them one by one. The most troublesome part was that during those two months, research reports, policy recommendations, and investment decisions produced based on the false files had already been distributed to more than a dozen departments.

Old Shen distinguished between two kinds of events. Slipping in a note is "session-level interference": the contaminant enters the reading table, affects that consultation, and clearing the table makes it clean. Changing the draft is "knowledge-base-level destruction": the contamination enters permanent storage, all subsequent consultations are contaminated, and the cleanup cost grows exponentially with the spread.

The archive later set two rules. First, all original files were compared with the backup repository once a month, with immediate alerts for any discrepancy. Second, researchers had to attach the file numbers and version numbers they cited when submitting reports, so problems could be traced back.

At an industry exchange meeting, Old Shen said: to deal with slipped-in notes, managing the access zone is enough. To deal with altered drafts, you need backups, comparison and traceability. The defense investment for these two problems differs by an order of magnitude, but many people treat them as the same thing.

After the meeting, someone asked Old Shen which attack was the most dangerous. Old Shen said: the most dangerous is when you think you are defending against altered drafts, but in fact you have only added protection against slipped-in notes. After half a year of calm, you suddenly discover thirty fake case files hidden in the warehouse—that is what it means to be too late.

Context contamination

During inference, attackers influence the model's output for that session by tampering with retrieval results or injecting malicious context. The contamination scope is limited to a single conversation window and does not change the model itself or the knowledge base.

Data poisoning

During training or data preparation, attackers inject malicious samples into training data, knowledge bases or vector databases. The contamination becomes fixed in model parameters or persistent storage, and all subsequent queries are affected.

Story Mapping

  • Tucking a note onto the reading table: context contamination—temporarily injected, affects the current query, and disappears once the context is cleared.
  • Re-filing after editing the draft: data poisoning — modifying persistent storage so that all subsequent queries are contaminated.
  • Cracking the case in one day with a slip of paper: the defense focus for context contamination is at the retrieval and access control level.
  • Two months of tracing back through revised drafts: cleaning up data poisoning requires backup comparison, version management, and traceability mechanisms.
  • Two rules: the context layer prevents injection, the data layer adds validation—both sides are needed but with different strength.

In the project

In security reviews, model the two kinds of contamination separately. The defense line for context contamination sits in retrieval result validation and prompt injection detection, and can be made into lightweight real-time filtering. The defense line for data poisoning sits in training data auditing, knowledge base version comparison, and ingestion validation, and requires offline processes and rollback capability. Do not combine threat surface assessments into one calculation—context contamination is high risk but quick to fix, while data poisoning is high risk and slow to fix, so their investment priorities and budget logic differ.

京ICP备2026014272号