← Back to Concept Contrast

Concept Contrast Fable

AI incident and safety incident response

Conclusion first:AI incidents are about model behavior going wrong — outputting harmful content, making wrong decisions, producing discriminatory results. Safety incidents are about systems being breached, data leaking, permissions being bypassed. Both kinds of incidents are serious, but the trigger conditions, detection methods and response processes differ.

AI incidents are about model behavior going wrong — outputting harmful content, making wrong decisions, producing discriminatory results. Safety incidents are about systems being breached, data leaking, permissions being bypassed. Both kinds of incidents are serious, but the trigger conditions, detection methods and response processes differ.

Fable

Two kinds of alarms in the emergency room

The clinic in the east of the city had just hung a bronze diagnostic mirror, while the medicine warehouse in the west often guarded against thieves climbing over the wall. The court physician managed two seemingly similar matters: one was how to limit losses after the mirror gave a misdiagnosis, and the other was how to investigate after the medicine warehouse was broken into and prescriptions were stolen. The new apprentice found it troublesome and merged the two ledgers into one, thinking that both were related to success and failure anyway.

At first no one objected. Both sides wrote it up as an incident log, the night watchman only locked the door and interrogated according to theft rules, and the shopkeeper, seeing fewer pages, even praised them for saving effort. But once peak season arrived, the old accounts suddenly could not be explained: was it that the previous stall had not finished, or that the next stall had not held the line? No one could answer.

They first added more seals and slogans and required everyone to write in more detail. As a result, the booklet grew thicker and disputes increased. When something went wrong, everyone could find a sentence to excuse themselves, but no one knew which end to save first.

The old steward called everyone into the courtyard and had them retrace the actual course of events. Whatever concerned the immediate action went into the first book; whatever concerned subsequent responsibility went into the second. The two books referenced each other, but were no longer mixed onto the same page.

Once, the bronze mirror pointed several patients to the same medicine. The pharmacy did not drop a single drug, yet the patients were nearly delayed. After separating them, many old conclusions changed: some activities looked busy but did not actually block trouble; some arrangements were unremarkable in normal times yet preserved the whole in a disaster.

To prevent newcomers from mixing things up again, the old steward had them each handle a small case. Only when the two ledgers could each answer their own questions and also connect with each other at the boundary would the small case pass. If someone still wrote the two matters on one page, they had to return to the scene and look again at who acted first and who was ultimately responsible; when it was unclear, it was better to ask once more than let a muddled account stay overnight; because on the day something truly went wrong, one missing line of distinction would make an extra team work in vain.

The court judge separated the two response paths: for a theft case, seal the door and check for traces first; for a misdiagnosis case, stop using the bronze mirror, review the medical records, and notify affected people first. Later, when newcomers entered the trade, the old steward had them memorize just one line: when a lock is broken, chase the thief; when a judgment harms someone, save the person first. Both are called incidents, but the order of handling must not be mixed up.

AI incidents

Harmful or unintended behavior produced by an AI system during operation, including erroneous output, discriminatory results, misleading model hallucinations, and AI systems going out of control. Its root cause is model behavior problems, not the system being breached externally.

Security incident response

A standardized response process for information system security incidents, including detecting intrusions, containing spread, removing threats, restoring systems and post-incident tracing. Its targets are traditional security incidents such as cyberattacks, data breaches and privilege abuse.

Story Mapping

  • Hospital-wide alarm system: traditional security response systems that monitor network attacks and system intrusions.
  • AI misdiagnosis reports did not trigger alarms: AI incidents fall outside the coverage of traditional safety monitoring.
  • AI incident response team: a dedicated team and process for handling abnormal behavior in AI systems.
  • Malicious prompts cause triage errors: the intersection of AI incidents and security incidents requires a joint response.
  • Joint exercise: the two response systems need clear boundaries, linkage rules, and command ownership.

In the project

When delivering AI systems to customers, add an AI incident response section to the security questionnaire. Ask clearly: who monitors abnormal model output? What is the rollback mechanism for misjudgments? How are affected users notified? The answers to these questions will not appear in a traditional security emergency plan. If your AI system has not yet established an AI incident response process, your team will be holding the security manual at a loss when the first error occurs.

京ICP备2026014272号