Concept Contrast Fable

Risk assessment and model risk assessment: one scans the whole house, the other specifically checks AI

Conclusion first:The project initiation phase usually includes a round of risk assessment, and when AI models are involved, model risk assessment is also mentioned. Many teams simply fold model risk assessment into general risk assessment and do them together, with the result that the general risk template cannot detect the hidden risks of the model, and the model-specific check cannot cover the chain effects at the business level.

The project initiation phase usually includes a round of risk assessment, and when AI models are involved, model risk assessment is also mentioned. Many teams simply fold model risk assessment into general risk assessment and do them together, with the result that the general risk template cannot detect the hidden risks of the model, and the model-specific check cannot cover the chain effects at the business level.

Fable

Factory safety inspections and specialized equipment inspections

A chemical plant has two sets of safety inspections each year. The first is a comprehensive safety inspection covering fire access routes, hazardous chemical storage, explosion-proof electrical equipment, and emergency plans, across more than twenty workshops plant-wide. The second is a special inspection of automated equipment, covering only the sensor accuracy, control logic redundancy, and failover mechanisms of the central control system. The two inspections differ in report format, inspector qualifications, and rectification procedures.

One year the factory launched a cost-cutting and efficiency drive, merging the two inspections into one annual major inspection. The safety team figured that since they were entering the workshop anyway, they might as well check everything in one trip. The automation equipment section was embedded into the comprehensive checklist as a small subsection, and the original four-hour specialized inspection was compressed into forty minutes. The inspector went down the automation equipment subsection of the comprehensive checklist, ticked a few items as operating normally, and left.

Three months later, a slight drift occurred in the reactor temperature control system—the sensor reading was five degrees lower than the actual temperature, and the control system triggered no alarm because the deviation did not exceed the safety threshold set in the comprehensive checklist. Twelve hours later the temperature deviation widened to fifteen degrees, and the system finally alarmed, but corrosion had already appeared on the inner wall of the reactor. Production was halted for three weeks for repairs.

The accident investigation found that the automated special inspection originally had its own independent threshold standards: a temperature-control deviation of more than three degrees had to be marked yellow, and more than five degrees had to be marked red and trigger the maintenance process. These standards were overridden when merged into the general inspection—the general inspection checklist covered fire safety and hazardous chemicals, and its thresholds were too coarse for precision control systems.

The factory restored the double-inspection system, but made one improvement: a cross-reference table was added to the first page of both inspection reports. The comprehensive inspection report stated which specialized inspection items were not covered this time, and the specialized inspection report stated the impact of this item's risk on the factory-wide comprehensive score. The two reports were not merged, but each could see the other's existence and scope of conclusions.

At the year-end safety committee summary, the factory director hung the whole-factory map and the workshop cross-section side by side on the wall. From then on, he no longer stuffed the cross-section into the whole-factory map, nor used the whole-factory map as a substitute for opening up the equipment and taking a look. The two maps had different scales, and if either replaced the other, information would be lost.

Risk assessment

A general risk identification, analysis and evaluation process covering all types of risk for an organization, project or system, focused on the overall risk picture and management priorities.

Model risk assessment

Specialized risk checks for AI or machine learning models, covering model-specific risk dimensions such as model bias, data drift, adversarial robustness, explainability and output safety.

Story Mapping

Comprehensive safety inspection covering fire and hazardous chemicals plant-wide: general risk assessment, a global picture spanning multiple risk types
Special inspection of automated equipment checks sensors and control logic: model risk assessment, focusing on model-specific technical risks
After merging into a large inspection, dedicated inspection shrank from four hours to forty minutes: nesting model risk inside general evaluation leads to insufficient coverage
The general threshold of a comprehensive inspection masks the precision standards of a specialized inspection: the two types of assessment use different scales, and merging them loses precision
Add a cross-reference table to the first page of both reports: keep the processes independent but establish a mutually visible relationship

Applying it to an Agent project

Risk assessment at Agent project kickoff is divided into two independent parts. General risk assessment uses conventional frameworks to cover global risks such as data privacy, business continuity, and third-party dependencies. Model risk assessment uses a model-specific checklist to examine hallucination rates, bias, adversarial attack surface, and output safety issues. The conclusions of the two reports can be consolidated into the same risk register, but the inspection process, inspection criteria, and inspector qualifications must remain independent.

Risk assessment Model risk assessment