← Back to concept explainers

Concept Fable

Task success rate

Conclusion first:Task Success Rate measures the proportion of times an Agent or automation system truly completes the user's goal on a given task. It cannot look only at call counts, response counts, or whether the process started; it must clearly define success conditions, acceptance evidence, and failure classification. In enterprise deployment, this metric directly relates to whether it can replace manual work, reduce rework, and support outcome-based pricing.

Task Success Rate · Metric

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

At the post station, only carriages that depart are counted, not those that arrive

The mountain road post station used to have only a dozen letters a day. The stationmaster considered the day safe if he saw whether the carriage left on time and whether the coachman returned.

Later the letters became business contracts, prescriptions, lawsuits, and urgent reports. The carriage still left on time, but recipients often said they had not received it, received it late, or received it with a page missing.

The stationmaster first posted the number of trips on the wall. The numbers looked better and better, and the coachmen were more willing to take short routes, because every trip counted as an achievement.

The shopkeepers began to complain: the post station looked busier, but the number of things actually accomplished had not increased. After one prescription arrived late, the patient's family member held up the good numbers and asked the stationmaster whom these numbers had actually helped.

The stationmaster also wanted to count how many li the horse ran. The groom therefore took detours to make up mileage; the ledger grew thicker, but delivery remained chaotic. The new accountant broke each letter from order intake to signed receipt into several cells: address recognition, sorting, loading, in-transit handoff, receipt confirmation, exception handling.

Only when the recipient confirms and the content is complete does it count as success. At first the numbers were ugly, and many problems that had been hidden by "went out" were exposed. A few months later, the number of outbound trips decreased a little, but far more letters were truly delivered.

The new bookkeeper also classified failures. Some letters had addresses recognized incorrectly, some were lost during handover en route, and some arrived with missing pages. Each type of failure corresponded to a different fix; all bad outcomes could not simply be stuffed into one bucket.

The stationmaster later still looked at the number of trips and the distance traveled, but they served only as auxiliary. What truly hung at the center of the wall was the proportion of complete deliveries and the reasons for failure. The post station finally measured itself by what it was hired to accomplish. Later, when newcomers were onboarding, the old masters no longer started with abstract terms, but had them review that failure: where intuition had been relied on, why the first fix was insufficient, and how the new arrangement kept risk in a place that could be checked. Only when they could explain these details clearly, and also point out when to follow the old way and when to escalate, were they considered to truly understand this approach. Later, they also had newcomers practice this approach: first let the newcomers do it the old way to see how errors emerged; then redo it under the new rules to see which step was blocked in advance and which step left evidence. If newcomers could only recite names, they would still misuse it in similar scenarios; only those who could state the applicable boundaries, costs, and remedies after failure were allowed to take on work independently.

Reveal

This story is about: task success rate

Task Success Rate measures the proportion of times an Agent or automation system truly completes the user's goal on a given task. It cannot look only at call counts, response counts, or whether the process started; it must clearly define success conditions, acceptance evidence, and failure classification. In enterprise deployment, this metric directly relates to whether it can replace manual work, reduce rework, and support outcome-based pricing.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Carriage leaving the gate: the Agent begins execution or generates a response
  • Letters fully signed for: tasks completed according to business objectives
  • Trip count and mileage: activity metrics that are easy to inflate
  • Break it into cells: phased acceptance criteria for task success
  • Exception handling: failure cause classification and remediation paths
  • Truly delivering more: business outcome improvement brought by automation

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

End-to-end evaluationPass rateOSWorldWebArena
京ICP备2026014272号