← Back to concept explainers

Concept Fable

Operational metrics

Conclusion first:Operational metrics are quantitative metrics used to manage Agents in production, such as completion rate, escalation rate, error rate, latency, cost, rework rate, first-pass acceptance rate, and incident rate. Once an enterprise Agent truly goes live, you cannot look only at call counts, conversation volume, or demo performance; it must be continuously measured, reviewed, and improved like an operable digital workforce. The value of Andy Ren Xu's Services-as-Software also depends on these metrics: customers buy deliverable outcomes, and the supplier must know whether outcomes are stable, whether costs are controllable, and which tasks still require human fallback.

Operating Metrics · Agent Operating Model

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The accounting office changed the ruler, and only then did the workshop know who had truly finished the work

The South Bank workshop used to look only at month-end revenue. For every table and chair sold, the accountant drew a horizontal line on the wall. The masters all knew their craft, and there weren't many customers, so although this number was crude, at least it let the owner know whether the workshop had a way to survive.

Later the workshop took on long-term orders from city firms. Tables and chairs were no longer sold one by one, but repaired, replaced, delivered, modified and inspected daily. Some people worked fast but reworked a lot, some were slower but got it right the first time, some pushed hard orders to others, and at month's end the revenue still looked good while the floor grew more and more chaotic.

The boss first added a new metric: how many wooden tags each craftsman completed per day. The tag count rose quickly, and everyone began breaking large jobs into small tags, rushing to do what could be done quickly and leaving the troublesome parts for last. The numbers on the wall looked better, but customer complaints did not decrease.

He also had the bookkeeper record work hours. As a result, the craftsmen began proving how busy they were. Some spent a whole day on bad material, and some counted waiting for parts as full work hours. The busyness was recorded in full, but customers still could not get usable tables and chairs.

When a major trading firm returned thirty chairs, the boss realized he had never truly managed delivery. He knew how much he had sold, how much he had made, and how long he had been busy, but he didn't know the first-pass completion rate, rework rate, reasons for delays, time spent waiting for approvals, material waste, or whether customers had accepted the goods.

The new accountant wiped away half the horizontal lines on the wall and re-divided them into several columns: order received, completed, first-pass acceptance, rework, overtime, escalation, cost, and incidents. Every wooden tag had to leave a status from entry to delivery, not just turn into a single revenue entry at month-end.

At first the masters felt these numbers were like surveillance. Two weeks later, they found the benefits were very direct: being slow was not a crime, being stuck waiting for materials would be seen; rework was not an argument, but a way to find which step went wrong; a master who handled few orders but took only high-risk ones would not be misjudged.

In the end the boss stopped asking who looked busiest. Every day he first looked at which work completed automatically, which work escalated to people, where rework happened most, and whether costs had crossed the line. For the first time the workshop could manage work in progress, not just wait until month-end to see success or failure.

Reveal

This story is about: operational metrics

Operational metrics are quantitative metrics used to manage Agents in production, such as completion rate, escalation rate, error rate, latency, cost, rework rate, first-pass acceptance rate, and incident rate. Once an enterprise Agent truly goes live, you cannot look only at call counts, conversation volume, or demo performance; it must be continuously measured, reviewed, and improved like an operable digital workforce. The value of Andy Ren Xu's Services-as-Software also depends on these metrics: customers buy deliverable outcomes, and the supplier must know whether outcomes are stable, whether costs are controllable, and which tasks still require human fallback.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Month-end revenue: looking only at final business results, lacking process management
  • How many wooden tags are completed each day: a single output metric, which easily encourages splitting orders and cherry-picking easy work
  • Timesheet: mistaking busyness for effective delivery
  • Thirty chairs returned: delivery failure caused by missing quality, rework, and acceptance metrics
  • Accepted orders, completion, rework, overtime, escalation, cost: Agent production operations metrics system
  • Missing materials become visible: metrics must distinguish capability problems, dependency problems, and approval problems
  • The boss looks at upgrade rates and rework rates: manage agents with Operating Metrics, not by gut feeling

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

KPISLAquality scoreproductivityincident rate

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号