← Back to concept explainers

Concept Fable

Model risk management

Conclusion first:Model risk management is the discipline of governing the risks arising from model errors, misuse, drift, limitations, assumptions and operational failures. MRM typically includes model inventory, risk tiering, model validation, approval, monitoring, change management, vendor model management, exception handling and periodic review. For LLM/Agent systems, it must also cover output unpredictability, tool execution risk, data leakage, bias, security and human oversight.

Model Risk Management, MRM · Model Risk Management

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The city no longer asks only whether the calculation is accurate

Jintai City had many bronze machines that made judgments. They decided loan rankings, warehouse restocking, complaint priorities and inspection routes. At first, everyone asked only one question: are the bronze machines accurate?

A few years later, there were more and more bronze machines. Some came from city craftsmen, some from foreign merchants, and some were modified from old rules. They affected more and more money, time, and people, yet there was no common way to govern them.

The official's naive fix was to require an accuracy note attached to every copper machine. The note looked professional but couldn't explain many things: whether data would expire, where the boundaries were, who could change parameters, who was responsible for errors, and when it must be decommissioned.

An automated ranking put high-risk customers into the fast track. The bronze machine was accurate on old samples but amplified errors on new business. The responsible department said the vendor had guaranteed it, the vendor said the usage scenario had changed, and the auditor said there was no independent verification record.

So the city lord established a Machine Risk Bureau. It did not build bronze machines, nor did it make every business decision on behalf of the business. Instead, it required that all important bronze machines be registered, classified, validated, monitored, and reviewed, with clear owners and exception handling. The Risk Bureau broke the problem down: What is the machine's purpose? What are its assumptions? Who might it harm? How large could the losses be if it errs? Is there human oversight? How are supplier dependencies managed? What is the exit plan?

From then on, high-risk copper machines could not go live on the developer's self-certification alone. They had to go through independent verification, launch approval, runtime monitoring and periodic review. Low-risk copper machines had a lighter process, but they still could not run without an owner. At first the business departments felt this added a layer of gatekeeping, but later they found it made it easier to explain to clients, regulators and the board why they dared to use it, where it could not be used, and how incidents would be handled.

The city lord later established a bronze machine registry. Each bronze machine recorded its purpose, origin, scope of impact, responsible person, review date, and decommissioning conditions; the more it affected people's livelihoods, the more carefully it was reviewed. The craftsmen initially found it troublesome, saying bronze machines only needed to be accurate. Until an old bronze machine pushed all the small merchants to the back in a new market—the accuracy slips still looked beautiful, but complaints piled up at the government office.

From then on, Jintai City no longer looked only at arithmetic scores. Whether a machine could explain itself, whether responsibility could be traced, and who would re-inspect when the environment changed—these mattered as much as accuracy in deciding whether a bronze machine could take to the streets. The city lord finally understood: machine risk management isn't about blocking machines from entering the business—it's about giving machines boundaries, evidence, and exit mechanisms when they bear real responsibility.

Reveal

This story is about: model risk management

Model risk management is the discipline of governing the risks arising from model errors, misuse, drift, limitations, assumptions and operational failures. MRM typically includes model inventory, risk tiering, model validation, approval, monitoring, change management, vendor model management, exception handling and periodic review. For LLM/Agent systems, it must also cover output unpredictability, tool execution risk, data leakage, bias, security and human oversight.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • The bronze machine that makes judgments: models and AI systems that influence business decisions
  • Accuracy note: naive patchwork that manages risk with a single performance metric
  • Vendors and departments pass the buck: unclear responsibility boundaries
  • Model Risk Office: MRM governance functions and control framework
  • Registration, classification, validation, monitoring, review: the model risk management lifecycle
  • Independent validation: model validation and second line review
  • Exit plan: rollback, decommission and fallback plan
  • Final insight: MRM makes models governable, explainable, and exitable under real business responsibility

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

model validationmodel monitoringmodel inventorychallenger modelSR 11-7

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号