← Back to concept explainers

Concept Fable

Round-by-round evaluation

Conclusion first:Per-Turn Evaluation evaluates an Agent or dialogue system at each turn of action, tool call, observation, and reply, rather than only looking at the final result. It is suited to long-chain tasks, customer service, code repair, and multi-tool workflows, and can pinpoint specific failed steps and measure instruction following, tool selection, state updates, safety, and progress quality.

Per-Turn Evaluation · Evaluation Methods

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The small scoreboard beside each move

When the chess academy evaluated apprentices, in the past it looked only at the final win or loss. The old players thought this was fair: when the board was put away, whoever won was better, and there was no need to ask much about the process.

Later, as games grew longer, the problem surfaced. Some played the first ten moves well but blundered through the middle game; others won in the end only because the opponent made a mistake. Looking only at the final position, the teacher couldn't tell skill from luck.

The director first had everyone play a few more games. With more samples, it was still impossible to see which step misunderstood the situation and which step was merely forced by the opponent. The apprentice also did not know where to make changes.

He also required post-match self-assessment. Apprentices often described lucky wins as brilliant moves and downplayed critical errors; the more beautifully the review was written, the less accurate the training became.

A coach placed a small scoreboard beside the chessboard. With every move, he recorded the goal at the time, the available actions, the reasoning, the change in position, and whether it was close to plan.

Review no longer looks only at the final outcome, but can see whether the third move misread the situation, whether the seventh move chose the wrong means, and whether the twelfth move ignored a threat. Some wins are rated low because they relied on luck; some losses keep a high score because every move was reasonable given the information at the time. The chess academy can finally train specific abilities, rather than only rewarding outcomes.

Later, the chess academy used the small scoreboard for training, not just for punishment. A certain apprentice always misjudged the goal in the opening, so he practiced openings; another always chose the wrong exchange in the middlegame, so he drilled the middlegame specifically.

The final score was still kept, because a game must be won to be complete. But the record of each move let the teacher know how victory or defeat had come about. The chess academy no longer compressed a game into a single number, but broke the long process into many small judgments that could be improved. Later, when beginners were initiated, the old masters no longer started with abstract terms, but had them review that mistake: where they had originally relied on intuition, why the first fix was insufficient, and how the new arrangement kept risk in a position that could be checked. Only when they could explain these details clearly, and also point out when to follow the old approach and when to upgrade, were they considered to truly understand this method. Later they also handed this method to newcomers for practice: first have the newcomer do it the old way, to see how errors arise; then redo it according to the new rules, to see which step was blocked in advance and which step left evidence. If a newcomer could only recite names, they would still misuse it in similar scenarios; only those who could state the applicable boundaries, the costs, and the remedies after failure were allowed to take on work independently.

Reveal

This story is about: round-by-round evaluation

Per-Turn Evaluation evaluates an Agent or dialogue system at each turn of action, tool call, observation, and reply, rather than only looking at the final result. It is suited to long-chain tasks, customer service, code repair, and multi-tool workflows, and can pinpoint specific failed steps and measure instruction following, tool selection, state updates, safety, and progress quality.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Overall win or loss: final task success rate
  • A small scoreboard at every step: round-by-round evaluation
  • Goals, reasons, and situation changes: turn-level traces
  • A lucky win: outcome metrics mask process problems
  • Wrong tool selection: failure at an intermediate step
  • Training specific capabilities: improving based on fine-grained feedback.

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

Conversation quality evaluationTurn-level scoringContext dependence

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号