← Back to Concept Contrast

Concept Contrast Fable

Operational metrics and task success rate: one looks at whether the machine runs well, the other at whether the machine does the right work

Conclusion first:Putting two reports on the same page makes it easy for the team to think they are looking at the same thing. Operational metrics tell you how well the system is running—response time, availability, throughput; task success rate tells you the proportion of user intents the Agent completes. If the system runs extremely fast but more than half of tasks fail, the green numbers on the dashboard become an illusion.

Putting two reports on the same page makes it easy for the team to think they are looking at the same thing. Operational metrics tell you how well the system is running—response time, availability, throughput; task success rate tells you the proportion of user intents the Agent completes. If the system runs extremely fast but more than half of tasks fail, the green numbers on the dashboard become an illusion.

Fable

The two leaderboards on the courier station wall

Two leaderboards were posted on the wall of the delivery station in the north of the city. The one on the left was called the "Operations Board," tracking each courier's pickup speed, average daily deliveries, and return processing time. The station manager looked at this board every day to assign performance, and whoever had the higher numbers got the bonus. Rankings on the Operations Board directly determined the monthly allocation.

The one on the right was called the “complaint board,” tracking customer complaint volume, misdelivery and missed delivery counts, and claim amounts. The stationmaster did not look at it much—because complaint data was only summarized at month-end, and the same few people were always at the bottom, so a mention at the meeting was enough. The two boards were on the same wall, but the attention they received differed tenfold.

During the Double Eleven peak, a courier named Lao Luo shot to first place on the performance leaderboard. His average daily delivery volume was significantly higher than his colleagues, and his pickup speed was far ahead. The station manager publicly praised him at the morning meeting and promoted Lao Luo's methods as a benchmark.

At the end of the month, something strange showed up in the summary data. Old Luo ranked first in deliveries, but also first in customer complaints. A closer look revealed that to increase speed, Old Luo habitually left large packages at the community gate instead of delivering them upstairs, and after signing on the recipient's behalf he did not notify them. The system records all showed "signed for," and the operations leaderboard was all green lights, but customer feedback on the complaint leaderboard exposed the true quality of completion.

The station manager brought this to the regional meeting for discussion. The regional manager drew two circles on the whiteboard: on the left he wrote "system-level metrics"—pickup time, number of deliveries, signature rate. On the right he wrote "task-level metrics"—the customer receives the package, the contents are intact, there are no complaints. The two circles overlap, but their areas are very different.

The station later changed its review rules. Rankings on the operations board no longer directly determined bonuses; instead, a threshold condition was added: only those with complaint rates below the regional average were eligible for ranking rewards. The operations board managed efficiency, the complaint board managed correctness, and neither could be missing.

In the first month of the new rules, efficiency temporarily dropped a little—couriers no longer dared to toss packages. But at quarterly settlement, the station's return rate and claims expenses fell by nearly half, and profit actually rose. In the year-end summary, the station manager wrote one sentence: a machine running fast is a necessary condition, but running correctly is the sufficient condition. Do not report the first as the second.

Operational metrics

Metrics that measure system operating state—latency, throughput, availability, error rate—focus on "whether the system runs well."

Task success rate

Measures the proportion of user intents an Agent completes — whether the task is correctly closed out and whether the result meets expectations, focusing on "was the job done right."

Story Mapping

  • Pickup speed and delivery volume on the operations board: operational metrics, system-level operating efficiency
  • Misrouted and missed deliveries on the complaint board and customer feedback: task success rate, whether user intent is correctly fulfilled
  • Old Luo ranked first on the operations leaderboard but also first in complaints: good operational metrics do not equal a high task success rate
  • Bonus plus complaint-rate threshold: operational metrics and task success rate assessed jointly, to prevent one-sided optimization

Applying it to an Agent project

The Agent's monitoring dashboard must have two independent sets of metrics. Operational metrics look at system health—API latency, Token consumption, error rate; task success rate looks at the business loop—whether user requests are correctly understood, whether Agent actions reach the correct result, and whether manual intervention and retry are needed. Set alert thresholds for the two sets of metrics separately. A system that is all green but drops into a pit on tasks is harder to discover than a system that is down, and the damage is deeper.

京ICP备2026014272号