← Back to concept explainers

Concept Fable

Step success rate

Conclusion first:Step success rate measures the proportion of intermediate steps executed correctly in a multi-step task. For enterprise Agents, it locates quality problems better than final success rate alone: whether the plan is reasonable, whether retrieval hits, whether the right tool is chosen, whether parameters are correct, whether permissions pass, and whether exceptions are handled. Step Success Rate is suited to being combined with traces, spans, error classification, and human review to find which link is dragging down overall delivery.

Step Success Rate · Evaluation

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The long road isn't just about whether you reach the destination

Baisha Station was responsible for delivering urgent documents to the military camp behind the mountain. In the past, the route was short, and as long as the courier finally handed over the letter, it counted as success.

Later, the route grew longer: changing horses, passing checkpoints, stamping seals, crossing rivers, finding guides. Sometimes the letter arrived in the end, but along the way it had taken detours, been stamped incorrectly, or nearly handed to the wrong person.

The postmaster's naive fix was to reward only final delivery. The couriers began desperately racing to the finish, and records along the way became fewer and fewer. The letter arrived, but no one knew which step had nearly gone wrong.

After a heavy snow, all three letters were late. The courier said the mountain roads were hard to travel, but the post station chief could not tell whether it was slow horse changes, the wrong checkpoint, the ferry not waiting, or the guide going the wrong way.

An old courier suggested breaking the long route into fixed checkpoints: whether the letter was picked up correctly, whether the route was chosen correctly, whether the travel documents were complete, whether the tools were used correctly, whether anomalies were reported, and finally whether it was delivered.

They no longer just made a single note at the finish line; instead, they stamped a small seal at every step. Each small seal had success, failure, retry, manual takeover and the reason. The post station also used the small seals to train newcomers. Newcomers did not have to wait for the whole journey to fail before knowing where they were weak—if they kept making mistakes at the second checkpoint, they practiced the checkpoint documents; if they were slow at crossing the river, they switched to the boat agreement.

Soon the post station discovered that the final success rate was masking a key problem: most failures happened at the second horse change; some couriers did deliver in the end, but often relied on acquaintances at the checkpoint to patch things up. The military camp also became more willing to trust the post station, because what they saw was not just the last letter, but the stability of every node along the route. The capability for long tasks is hidden in these continuous small successes.

There was now a basis for repairing roads, replacing horses, and training newcomers. The courier station chief could also judge that a route was not bad as a whole, but that certain steps were fragile. In the end, he understood that multi-step work depends on whether each step stands firm. Looking only at the endpoint mistakes luck and remediation for capability. Later, the manager wrote this lesson into the daily rules: first look at the scenario, then look at the boundaries, and finally look at who can take over when something fails. Only when all of these are clear does a beautiful demo become deliverable work.

Reveal

This story is about: step success rate

Step success rate measures the proportion of intermediate steps executed correctly in a multi-step task. For enterprise Agents, it locates quality problems better than final success rate alone: whether the plan is reasonable, whether retrieval hits, whether the right tool is chosen, whether parameters are correct, whether permissions pass, and whether exceptions are handled. Step Success Rate is suited to being combined with traces, spans, error classification, and human review to find which link is dragging down overall delivery.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Long-distance letter delivery: multi-step Agent tasks
  • Changing horses, passing checkpoints, stamping seals, crossing rivers: intermediate steps such as planning, retrieval, tool calls and permission checks
  • Reward only final delivery: look only at end-to-end success rate
  • Stamping every step: scoring each step in the trajectory
  • Retries and human takeover: recovery mechanisms and human fallback
  • The second horse-switching problem: locating the bottleneck through step metrics
  • Final insight: Step Success Rate breaks down the quality of complex tasks into improvable stages

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

trajectorytool calls

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号