← Back to Concept Contrast

Concept Contrast Fable

Choosing the right tool and completing the task are two different metrics

Conclusion first:Tool-call accuracy measures whether the model picks the right function and fills in the right parameters when calling a tool. Task success rate measures whether the user's request is ultimately completed correctly. Picking the right tool doesn't mean the task is done—the tool call is just one link in the task chain; after the call there's still whether the parameter values are correct, whether the call order makes sense, and whether the results are used correctly.

Tool-call accuracy measures whether the model picks the right function and fills in the right parameters when calling a tool. Task success rate measures whether the user's request is ultimately completed correctly. Picking the right tool doesn't mean the task is done—the tool call is just one link in the task chain; after the call there's still whether the parameter values are correct, whether the call order makes sense, and whether the results are used correctly.

Fable

A runner who got the menu number right but delivered the wrong dish

The grand restaurant in the east of the city added a serving window between the main hall and the back kitchen, with a runner standing at the window waiting for menus. His job was: guests order, he reads the menu number, passes it through the window into the back kitchen, then carries the dishes that come out to the corresponding table. To assess the runner, the manager installed a record board at the window that tracked only one thing—whether the menu number was passed to the back kitchen correctly.

In the first month, the record board showed a delivery accuracy rate of 97 percent. The shopkeeper praised the clerks at the regular meeting. But the number of customer complaints doubled. The complaints were concentrated in the serving step; the order-passing step actually had few problems—the clerks delivered sweet-and-sour fish to the table that had ordered steamed fish, took Table 3’s soup to Table 8, and once even failed to hold the plate steady, spilling half a dish onto the floor.

The shopkeeper laid the data from the record board and the complaint list side by side on the table and studied them all afternoon. The record board said the correct-delivery rate had risen from ninety percent to ninety-seven percent—the assistants were indeed getting better at picking menu numbers. The complaint list mentioned wrong tables, dropped plates and spilled soup—none of which had anything to do with menu numbers. The shopkeeper understood one thing: delivering the right menu is the first step, but the right menu is no guarantee of the right table.

The restaurant changed two things. First, it kept menu-number accuracy in the assessment table, but only as an intermediate metric. Second, it added a customer satisfaction tracker — after each table finished eating, guests checked satisfied or unsatisfied, which was compared against the menu number to calculate the proportion of guests who ended up satisfied after the menu was delivered correctly. That is the true metric.

The errand boy began to notice a pattern: the more complex the menu number, the more time he needed to verify it, and the higher his delivery accuracy became—because he looked a few seconds longer. But the ones he got wrong most often were familiar old menus—he glanced and handed them over, the number was right, but he did not notice that the old menu had changed its preparation last week, sweet-and-sour fish had a different side dish, and the customer did not recognize the dish that was served. The right tool was selected, but the parameter value was wrong.

The kitchen later split the metrics into three gates: the first gate looked at the accuracy of passing along menu numbers, the second gate looked at whether the dish matched the guest when the number was correct, and the third gate looked at guest satisfaction. Each gate had corresponding improvement measures—the first relied on menu training, the second on checking the dish-serving process, and the third on direct guest feedback.

A small line was later added to the sign: whether the food is served correctly is not only about the little window where orders are handed in. The empty plates on the guest's table are the real bill.

Tool call accuracy

Measures the proportion of times a model selects the correct function name and fills in the correct parameter format when it needs to call a tool. It only evaluates the correctness of tool selection, not the completion quality of the entire task. It is an intermediate node metric in the task chain.

Task success rate

Measures the proportion of user requests that are ultimately completed correctly. It covers the full chain from understanding the request, selecting tools, invoking tools, using tool results, to generating the final answer. It is an end-to-end user experience metric.

Story Mapping

  • Handoff accuracy on the record board: tool-call accuracy, measuring only tool selection and parameter format
  • Customer satisfaction tracking: task success rate, covering the end-to-end chain from request to result
  • Passing it to the right place does not mean serving it right: choosing the right tool but getting parameter values or call order wrong can cause task failure
  • A 97% tool-call accuracy rate plus doubled complaints: a typical scenario of high tool-call accuracy but low task success rate
  • Three gates: a complete evaluation system needs both intermediate metrics and end-to-end metrics
  • Familiar with the menu but still makes mistakes: the tool is chosen correctly but the context change is not noticed, and an incorrect parameter value causes failure

Applying This to Agent Quality Evaluation

When building an Agent evaluation system, keep tool-call accuracy as a debugging metric—it helps you locate whether the problem is tool selection or a later step. But you cannot use tool-call accuracy to replace task success rate when reporting to clients or business stakeholders. Common gaps between the two include: wrong parameter values, wrong call order, and tool results misinterpreted by the model. When task success rate drops, it's advisable to trace back to tool-call accuracy for attribution, but never use the former alone to replace the latter in go-live decisions.

京ICP备2026014272号