Concept Contrast Fable

Evaluate the entire task path and evaluate each round of response

Conclusion first:In multi-step tasks, looking only at politeness and accuracy in each turn misses whether the whole path was completed, whether it took a detour, whether anything was dropped. Trajectory evaluation answers whether the task was delivered; turn-by-turn evaluation answers whether each exchange was appropriate.

In multi-step tasks, looking only at politeness and accuracy in each turn misses whether the whole path was completed, whether it took a detour, whether anything was dropped. Trajectory evaluation answers whether the task was delivered; turn-by-turn evaluation answers whether each exchange was appropriate.

Fable

Two examiners

The firm hired a group of couriers responsible for delivering waybills, contracts, and samples to various cities. The manager invited two examiners to evaluate the couriers. The first sat behind the counter and had each courier report at every station. He only evaluated whether each report was clear, polite, and used the correct titles. Those who reported beautifully got high scores; those who stammered got low scores.

The second examiner did not sit at the counter, but followed the errand runner along the entire route. He did not look at whether the report was beautiful, only at whether the goods list was complete at departure, whether shortcuts over the mountains were taken midway, whether items were missing on delivery, and whether the customer signed for receipt. Only after the whole route was walked did he give a score.

When the first evaluation results came out, the two examiners' scores did not match at all. The first-place finisher with the polished report had lost two shipments on the road, taken three detours, and delivered half a day later than agreed. A runner with a halting report had complete shipping documents, the shortest route, and a customer who signed on the spot.

The shopkeeper looked at the two evaluation systems side by side and found each had its uses. The counter examiner could see whether a runner could communicate with people and explain clearly when encountering surprises; the road-following examiner could see whether the task was truly completed and whether there were hidden risky actions in between.

The firm later used two sets of scores together to judge people. A runner who dropped goods on the road but reported clearly would lose path points with the road examiner, while the counter examiner would give communication points. With both signals in parallel, the manager was no longer dazzled by polished reports, nor would he reject a capable delivery person over one clumsy remark.

The new automatic errand carts made assessment more complex. The carts never reported back, but every trip automatically recorded the full route. Someone proposed scoring only on routes, but once a cart waited half an hour at a customer's door without saying a word, and the customer thought no one had come and left. The manager realized that scoring only on routes would miss every step that required communication.

Finally, the manager drew two columns on each evaluation form: was it delivered this trip, and how was each segment handled. If it was delivered but problems occurred along the way, inspect the intermediate steps; if the middle looked good but it was not delivered, inspect the endpoint of the path. Only after both columns were filled did he know whether the runner was worth continuing to use.

Later the examiner pinned the two papers together: answering a single step well only earns a small score; only completing the entire assignment, handing off without error, and not taking the wrong route counts as truly passing.

Trajectory evaluation

Evaluate the entire process by which an agent completes a task, focusing on whether the path is correct, whether the steps are necessary, whether there are errors or redundancies along the way, and whether the goal is ultimately achieved.

Round-by-round evaluation

Evaluate the quality of an agent's single-turn response in each interaction, focusing on whether the answer is accurate, relevant, safe, and compliant with tone and format requirements.

Story Mapping

Counter examiner: round-by-round evaluation, looking only at the communication quality of each interaction
Route examiner: trajectory evaluation, focusing on the full task path and final result
Beautiful report but lost items: good single-turn performance but task not completed
Stumbling report but delivered: poor single-turn performance but successful task delivery
Automatic errand cart: pure trajectory scoring misses steps that require interactive judgment

Applying this to the Agent evaluation system

For multi-step tasks such as customer service, ticket handling and data queries, the evaluation system must include both trajectory evaluation and turn-by-turn evaluation. Trajectory looks at whether the task is closed-loop, whether the steps are reasonable, and whether there is unauthorized or repeated invocation; turn-by-turn looks at whether each answer is safe, accurate and appropriate. Evaluating only single turns lets agents that can chat but cannot get things done score high.

Trajectory evaluation Round-by-round evaluation