← Back to concept explainers

Concept Fable

Capability assessment

Conclusion first:Capability Evaluation is itemized, reproducible testing of a model's or Agent's specific capabilities, such as reasoning, retrieval, tool use, long context, collaboration, refusal, safety boundaries, and business task completion. It is closer to enterprise deployment decisions than broad benchmark scores, because enterprises need to know which capabilities the system is usable for and in which scenarios humans or controls must be added.

Capability Evaluation · Evaluation methods

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

That entrance test in the craftsmen's guild that did not look at slogans

It used to be simple for the master craftsman to recruit apprentices. If a child could recite the tool names and hold a hammer steady, he could enter. The shop was small, the master watched every day, and whatever was lacking was taught slowly.

Later the city received a palace order, and carpenters, stonemasons and lacquerers had to work together on complex tasks. Apprentices who could recite tool names could not tell on site whether to measure the wall first or cut materials first.

The chairman first added a longer questionnaire. The apprentices memorized it more thoroughly and answered faster, but when they reached a real construction site, some still reversed the order of beams and columns.

Someone suggested seeing who could work fastest. As a result, the apprentice skipped checks in order to be fast and hid small mistakes inside a large project. The speed looked good, but rework was more expensive.

An old master changed the exam into several small rooms: reading drawings, selecting materials, collaboration, handling surprises, and explaining trade-offs. Each room had a clear passing line, and also recorded at which step failures occurred.

For the first time, the apprentices discovered that they were not simply "able" or "unable," but stable in some capabilities and fragile in others. The master also knew which lesson to reinforce.

After the palace order began, the chairman assigned only people who passed the corresponding small room to the corresponding process. No one used a single total score to explain all capabilities anymore.

The craftsmen later left a rule: to trust that a person can take on big jobs, you first need to know which small jobs he is truly reliable at. Later, an even trickier job came along. On the surface it looked like just an old problem magnified, but in reality it forced out all the gaps hidden in the process: who takes over first, who can make changes, how to return to the previous step when something goes wrong—places that used to be glossed over by acquaintances' tacit understanding now all had to be made clear. The person in charge no longer urged everyone to be more careful, but instead walked the site again. He had people record every handover, every failure, and every remedy—which places were merely slow, which places would harm the results, which places would be hard to trace back once wrong. After the new rules were rolled out, the first few days were not smooth. Some complained about an extra inspection, some felt their old experience was being offended. But when the next anomaly appeared, for the first time everyone could follow the traces to find where the problem occurred, instead of gathering around the results and guessing at each other.

Reveal

This story is about: capability assessment

Capability Evaluation is itemized, reproducible testing of a model's or Agent's specific capabilities, such as reasoning, retrieval, tool use, long context, collaboration, refusal, safety boundaries, and business task completion. It is closer to enterprise deployment decisions than broad benchmark scores, because enterprises need to know which capabilities the system is usable for and in which scenarios humans or controls must be added.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Memorizing tool names: looking only at knowledge or general leaderboard scores
  • Palace orders: complex enterprise tasks
  • A few small rooms: evaluation sets split by capability and by scenario
  • Define the pass line: acceptable acceptance criteria
  • Record failed steps: pinpointing capability gaps
  • Assign people by process step: configure Agent responsibilities by capability boundaries

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

Capability boundaryTask success rateTool use evaluation

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号