← Back to concept explainers

Concept Fable

Token budget

Conclusion first:A token budget is the upper limit of tokens allocated to a task, request, user, or workflow. It controls prompt length, retrieved context, tool output, model reasoning space, and answer length.

Token Budget · Cost / context

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The clerk issues only one bag of lamp oil per trial

The county office reviewed cases by going through files at night. In the early years there were few cases, so the clerk could light as many lamps as he wanted, and the oil store was never tight.

Later, commercial disputes, land deed controversies, and letters from afar all poured in at once. Every case claimed to be complex, and the clerks hauled old files, new evidence, and circumstantial proof onto the table. The lamp oil quickly ran dry.

The county magistrate first ordered fewer case files to be reviewed. The result was that simple cases were fine, but complex cases missed key evidence and were overturned after judgment.

He also distributed oil by person, one bucket per clerk each day. Some used the oil on easy cases, some, to be safe, kept even small cases going until dawn, while truly urgent cases ran out of oil.

A clerk changed to issuing oil pouches by case: ordinary cases got small pouches, difficult cases got large pouches, appeals had a separate reserve; exceeding the oil pouch required explaining why the lamp should keep burning.

The clerks began deciding before opening a case which files were worth reading, which needed only a summary, and which evidence must be kept for final verification. A few months later, the county office no longer treated oil bags as punishment. It was like planning before a trial: where the limited light shines determines which facts can be seen clearly. Without a budget, diligence will burn down the entire oil depot.

The oil did not suddenly increase, yet the yamen no longer collapsed at month's end. Complex cases could get enough lamp oil, and small cases would not unconsciously burn through the whole store. The chief clerk also set up a reserve oil cabinet, kept specifically for suddenly emerging new evidence, temporary appeals, and cases the county magistrate ordered re-reviewed, so that all the scribes would not burn through the oil from the very start.

The county magistrate understood that limited lamp oil and manpower could not be managed by complaining afterward; they had to become a budget before each case began. The chief clerk also prepared trade-off methods for the clerks. Case files should first be checked by table of contents, old evidence by summary, and key clauses left for final verification; if more lamp oil was needed, it had to be explained which judgment the additional oil would buy. During reviews, they would spot-check several old cases to see whether the size of the oil pouch matched the complexity of the case, avoiding excessive page-turning for simple cases while difficult cases went dark at key points. The clerks also learned to first ask what light was needed to decide the case, rather than bringing all files under the lamp.

Reveal

This story is about: Token budget

A token budget is the upper limit of tokens allocated to a task, request, user, or workflow. It controls prompt length, retrieved context, tool output, model reasoning space, and answer length.

In enterprise Agent systems, token budget is the control plane for cost, latency, context discipline, and service gross margin. Without a budget, the model easily turns cheap tasks into expensive ones.

Metaphor mapping

  • Lamp oil: usable tokens and related inference costs
  • Each case review: one LLM/Agent task
  • The Case File: Context, Retrieved Content and Tool Output
  • Distributing oil by person: crude global quotas
  • Issue oil pouches by case: allocate token budget by task/workflow
  • Overage notes: budget overflow strategy and cost governance

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

context windowcost control
京ICP备2026014272号