Concept Contrast Fable

Trajectory evaluation and step success rate

Conclusion first:Step success rate looks at whether each step was done correctly—each step is scored independently, and eight out of nine steps correct is eighty-nine percent. Trajectory evaluation looks at the direction of the whole run—nine steps all correct but the last step jumping into a dead end means the entire trajectory failed. Reporting only step success rate in Agent evaluation is a common illusion: every step was taken correctly, but in the end the customer's problem was not solved.

Step success rate looks at whether each step was done correctly—each step is scored independently, and eight out of nine steps correct is eighty-nine percent. Trajectory evaluation looks at the direction of the whole run—nine steps all correct but the last step jumping into a dead end means the entire trajectory failed. Reporting only step success rate in Agent evaluation is a common illusion: every step was taken correctly, but in the end the customer's problem was not solved.

Fable

Checking road signs and looking at the destination

The escort agency trained new drivers with two kinds of tests. Master A stood at every intersection to see whether the driver turned when he should; Master B sat at the destination to see whether the driver actually delivered the goods.

In the first month, Master A was very pleased: out of twenty intersections, cart drivers took the correct route on average nineteen times. The escort leader was ready to declare the training a success. But Master B brought out the ledger: one fifth of the goods never arrived.

The escort leader did not believe it, so he pulled several complete journeys. One driver got the first nineteen steps right, but missed the bell at the last fork and took a forty-li detour; another looked reasonable at every single step, but when connected they formed a zigzag that shook the fragile porcelain to pieces.

Master A said, I can point out which step went wrong. Master B said, I can judge whether the whole trip succeeded. Neither was wrong; looking at only one side would lead to misjudgment. The escort agency redesigned the assessment. Step-by-step checks were kept, to locate the bad intersection; whole-trip assessment was also kept, so that if the goods did not arrive or were damaged, the entire trip was recorded as a failure. Then they went back into the route to find which steps caused the failure.

After the change, the escort agency discovered an old problem: some drivers followed the rules at every intersection, yet wasted too much time between waiting, U-turns, and lane changes. Single-step scores were beautiful, but the full route failed.

The convoy leader first scolded those cart drivers, saying their judgment at intersections was not steady enough. The drivers aggrievedly spread out the route permits: they had followed the rules at every intersection, but at the fifth segment they encountered a collapsed bridge, and after detouring no one checked the subsequent choices; one cart had gone correctly through nineteen intersections, yet finally stopped in front of the warehouse gate in the next town.

So the escort agency added a full-journey review. The crossroads still had to be watched, but they also had to see how the driver discovered the collapsed bridge, how he detoured, how he protected the goods, and whether he informed the consignee in time. Master A's signboard could spot small mistakes; Master B's ledger could see whether the whole trip succeeded. Counting only each step would mistake a beautiful halfway point for success.

The escort leader finally said: taken apart, you can find the parts; put together, you know whether this cart can finish the journey. To evaluate a continuous job, you must look at each step and at the whole path.

Trajectory evaluation

A method for evaluating Agent performance using the entire execution trajectory as the unit, focusing on whether the final task was completed, whether the path was reasonable, and whether there were ineffective loops. It measures "whether the result is right over the whole journey."

Step success rate

An evaluation method that measures accuracy in units of single-step operations, focusing on whether each step is executed as expected. It measures "whether each step is correct on its own" and does not consider the combined effects between steps.

Story Mapping

Group A checks the intersection reminders segment by segment: step success rate — independent scoring for each single step.
Group B follows the vehicle to see whether it reaches the destination: trajectory evaluation — overall task completion.
Nineteen steps correct and one wrong leads to a detour: a high step success rate masks a fatal error in a critical step.
Individually correct steps joined into a zigzag: combinatorial errors can only be exposed by trajectory evaluation.
Break it apart to locate, put it together to judge: the complementary relationship between two evaluation methods.

In the project

Agent evaluation should look at at least two layers at the same time. Step layer: tool selection accuracy, parameter filling accuracy, and output format compliance rate, used to locate specific failure points. Trajectory layer: task completion rate, number of invalid loops, and number of steps needed to reach the goal, used to judge overall quality. When cross-analyzing the two layers of data, watch for one dangerous signal: high step accuracy but low trajectory completion rate. This means the Agent walked in the wrong direction on the right path—the single-step logic is fine, but the global orchestration is flawed.

Trajectory evaluation Step success rate