← Back to concept explainers

Concept Fable

Prefix caching

Conclusion first:Prefix caching reuses the already-computed KV cache for the same prompt prefix, such as system prompts, shared policy text, product documents or long reference context. It caches the computation results of the shared context, not the final answer directly.

Prefix Caching · Inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The pier's main office no longer recopies the same opening letter every day

The seaside merchant guild wrote replies to out-of-town branches every day. Each letter had to begin with the same set of rules: which goods could be bought on credit, which customers required review, which promises could not exceed authority. The old clerk copied it word by word, while young assistants queued nearby waiting for instructions.

At first this wasn't a problem. Later, as the branches multiplied and the letters grew thicker, the real issue still hadn't been written before half a day had been spent on that fixed opening. Urgent orders got no reply, and ordinary ones were dragged into the evening.

The shopkeeper first had the clerk hire two more people, then shortened the opening letter. Newcomers copied faster, but still had to start from the first sentence; once the opening changed, everyone had to memorize it again. It looked like a few lines were saved, but the team was still jammed at the entrance.

Someone suggested printing the opening letter as a booklet and handing one to everyone who came. But when the clerk approved letters, he still had to read the booklet's contents back into his head. What truly consumed effort was not paper and ink, but reprocessing the same background passage for every letter.

The accounting office later made a hidden compartment. For any completely identical chamber-of-commerce opening, the same set of permissions and the same reference ledger, the senior clerk would handle it once and leave a reusable draft of annotations. If the next letter's beginning was completely identical, he would continue writing from after the draft.

He did not allow all letters to reuse the same draft. If the inn's ledger changed, permissions differed, or new clauses were inserted into the opening, it had to be annotated again. Reuse happened only when the earlier parts were truly the same and the boundaries were also the same.

A few days later, the queue was noticeably shorter. The clerk focused his energy on the truly different tail of each letter: whether this debt should be approved, whether this customer crossed the line, and whether this delivery needed a supervisor's signature.

The shopkeeper finally saw clearly that what was saved wasn't paper, but the time spent reprocessing the same opening passage. The longer the fixed prefix and the more incoming letters, the more valuable the hidden compartment; once the prefix differed, rushing to reuse would drag old rules into a new case. Later the guild divided common openings into several categories: ordinary credit, VIP review, cross-port delivery, dispute handling. Each had its own template and invalidation conditions. The old clerk said the hidden compartment saves effort only on the premise that you know which passage is truly the same passage; if you only look at similar openings, you'll copy someone else's rules into this letter. The young apprentice also learned to verify the prefix first, then enjoy the time saved. As long as even one part of the prefix differed, he'd rather re-approve than treat the old template as a master key.

Reveal

This story is about: prefix caching

Prefix caching reuses the already-computed KV cache for the same prompt prefix, such as system prompts, shared policy text, product documents or long reference context. It caches the computation results of the shared context, not the final answer directly.

In enterprise Agents, it can reduce prefill computation, improve time to first token and throughput, and lower costs. But the cache must be isolated and invalidated by model, tokenizer, prompt bytes, tenant, permissions, and version to avoid reusing expired or unauthorized context.

Metaphor mapping

  • Chamber of Commerce opening letter: system prompt, policy description, or long context prefix shared by multiple requests
  • The old clerk recopied every time: every request re-executes prefill
  • Annotated draft: reusable KV cache prefix
  • Reuse only when prefixes match: the hit conditions of prefix cache
  • Ledger and permission changes must be recalculated: tenant isolation, versions, permissions, and invalidation policies
  • Shorter queues: TTFT, GPU usage and request latency drop
  • Final insight: the value of Prefix Caching is not caching answers, but caching the shared context that has already been computed

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

KV cacheprefillsystem promptthroughput

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号