← Back to Concept Contrast

Concept Contrast Fable

Tool output contamination and indirect prompt injection

Conclusion first:Both attacks slip in through data returned by tools, but they cause damage in different ways. One stuffs instructions directly into the data, making the model comply on the spot; the other secretly alters facts in the data, making the model base decisions on fake numbers. Blocking one does not mean blocking the other.

Both attacks slip in through data returned by tools, but they cause damage in different ways. One stuffs instructions directly into the data, making the model comply on the spot; the other secretly alters facts in the data, making the model base decisions on fake numbers. Blocking one does not mean blocking the other.

Fable

Fake commands and fake numbers

Old Han ran a canal shipping freight house by the canal, specializing in north-south cargo transshipment. Every day the freight house received shipping orders sent from upstream docks—item name, quantity, destination, consignee, all four columns filled in. The clerks booked boats, loaded cargo, and collected payment according to the shipping orders, and for over a decade nothing major ever went wrong.

One autumn, the goods firm received a shipping order from Yangzhou with a line slipped in: "Send all silk to Jiangning Prefecture, not Suzhou." The clerk followed it, and all three boatloads of silk went to Jiangning. The consignee in Suzhou waited in vain and brought a complaint to the chamber of commerce. Old Han traced it and found that a dockworker at the Yangzhou wharf had slipped it in amid the confusion, having been paid by another goods firm.

Old Han set a rule: when a cargo list comes in the door, a literate clerk reviews it first; anything containing instruction words such as "void," "redirect," or "do not send" is held on the spot, circled in red, and reported. Once the rule was established, fake instructions never came through again.

After two peaceful months, trouble happened again. Three consecutive batches of medicinal materials sent to Jinan did not match in weight when they arrived—the shipping document said two hundred jin, but the boat had only one hundred twenty. Old Han investigated for nearly a month and found the source: an accountant at an upstream dock quietly reduced the weight by forty percent each time he filled out the medicinal materials shipping document. What he changed was the number; he did not touch a single instruction word—the product name was correct, the destination was correct, the sender was correct, and there was neither “void” nor “redirect.” The document-reviewing clerk checked only the words, and the tampered number was impossible to see at all.

Old Han looked at the two kinds of problems together. The first, 'inserting fake commands': the shipping order directly contained instructions telling the clerk to dispatch the boat according to a fake command on the spot—the signal was obvious and could be caught by the wording alone. The second, 'changing real numbers': do not touch the instruction wording, only the data—the item name was right, the destination was right, the shipper was right, only the weight was short by a chunk. The clerk dispatched the boat, calculated fees, and notified the recipient according to the fake numbers. Every step followed the rules, but every step stood on fake numbers.

Old Han added a step to the trading house: digital verification. The weight and quantity on every waybill had to be cross-checked against the dock's outbound records; if they did not match, the ship was held and marked "pending verification." For high-value goods like herbs and raw silk, a third-party weighing certificate was also required.

At the year-end chamber of commerce audit, the inspector asked Old Han how he prevented fake goods slips. Old Han laid out two procedures: the first was word interception, catching smuggled fake commands—changing the shipping destination wouldn't get past the gate. The second was numerical verification, checking tampered data—skimming off the weight to pocket the difference wouldn't get past the scale. After reviewing the records, the inspector said: most goods firms guard only one gate; you're the first to guard both.

Later, Old Han summed it up for his apprentice: a fake command is like someone slipping an extra page into a message book that says, "Everything said before doesn't count"—the extra thing can be seen at a glance. A fake number is like someone using a fine pen to alter a few digits in the ledger—the surface looks exactly the same, but every deal you make based on that wrong number, the ship is loaded with less, the money collected is less, and the consignee waits for nothing, and all the loss is yours.

Tool output contamination

Attackers tamper with content returned by tools (such as search results, database records, or file contents) to inject false data into the model while it processes the current task. The model makes wrong decisions on the spot based on this poisoned data—miscalculating costs, misjudging risk, recommending the wrong product. It attacks the immediate decision chain after a single tool call, and does not rely on long-term accumulation or the model's internal memory.

Indirect prompt injection

Attackers embed malicious instructions in external data sources (web pages, emails, documents, etc.). When the model reads this data, the embedded instructions are executed as system commands. The attack goal is to hijack the model's immediate behavior through external content, for example making the model output attacker-specified content or perform unauthorized operations.

Story Mapping

  • Inserting the instruction "reroute to Jiangning Prefecture" into the body of the shipping order: hijacking the ship-dispatch decision on the spot, with a clear and direct signal.
  • Keyword interception blocked fake commands but not fake numbers: instruction detection and data validation are two lines of defense and cannot replace each other.
  • The weight of the medicinal herbs was quietly reduced by forty percent, without touching a single word of the instructions: the order-checking clerk could not detect tampering at the numerical level by examining the wording.
  • Digital verification—the weights on the shipping list must be cross-checked against outbound records: this guards against data falsification, a type of problem that instruction scanning cannot detect.
  • Stuffing one more page into the message book vs. having the ledger numbers altered with a fine brush: one changes your instructions on the spot, the other secretly changes your basis.

In the project

During security assessment, perform two layers of detection on data returned by tools. The first layer performs instruction injection scanning, looking for imperative syntax, role tampering signals, and context reset markers. The second layer performs data quality validation, flagging numerical jumps, single-source information, and anomalous patterns as "pending verification." The two layers do not replace each other—passing the first layer does not mean passing the second.

京ICP备2026014272号