← Back to concept explainers

Concept Fable

KV cache

Conclusion first:KV Cache stores the Transformer attention keys and values computed for previous tokens. During autoregressive decoding, the model can attend to the cached history instead of recomputing keys and values for the entire context at every generation step. It is essential for practical LLM serving, but it consumes GPU memory and becomes a major constraint for long context, high concurrency, and multi-tenant Agent workloads.

KV Cache, Key-Value Cache · LLM runtime / optimization

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

At the long-table meeting, someone finally remembered what had already been said

The mountain temple's scripture library originally ran on a set of familiar methods. When the preaching monk began, he always built up slowly from the beginning of the scroll, and when answering questions at the old pace he rarely slipped up; the audience was also willing to follow along and revisit the earlier text.

Later the matter grew larger. The long scroll still needed continued explanation into the small hours, and the urging voices lined up from early morning to evening. The newcomers thought this was a quantity problem—just send more people and add more hours; the veterans, however, saw that many details had already crossed the boundary the old methods could bear.

The person in charge tried the first fix: reread from the beginning of the scroll for every question. For the first few days there was indeed improvement, the line got shorter, and outsiders felt this craft had finally caught up with the new scene.

But trouble quickly changed shape. Some people skipped inspections to rush progress, some forced unsuitable work into old processes, and the most skilled people were dragged down by trivial matters. Errors were not always visible on the surface and often only emerged after delivery. The old ledger could still explain where things came from, but it could not explain how to change for the next step.

The person in charge added more rules, requiring everyone to sign more, report more, and add another layer of protection. On paper it looked thorough, but on site it was slower; the craftsmen were busy proving they were not wrong, while the real cracks continued to widen in place. Onlookers only saw results fluctuating between good and bad, and people on site found it increasingly difficult to judge whether to save speed, quality, or responsibility first. Everyone felt they had done a little more, yet the whole thing did not become more stable because of it.

A silent old craftsman laid several failures on the table. He blamed no one, only had everyone look at the same detail: the new restriction had never entered daily practice, and only suddenly appeared before delivery. He laid the several reworks out in the order they occurred, and only then did everyone discover that mistakes were always amplified at the same corner.

So the workshop adopted a new rule: keep the passages already discussed and their corresponding annotations in a wooden box at hand. At first everyone found it uncomfortable, as if their hands and feet were tied; but after a few rounds, the rough shortcuts were blocked, and the truly sound moves gradually remained.

When facing the same pressure again, delivery did not become fancy, but it was clearly more reliable. The preaching monk finally understood: remember the clues already verified earlier, so later responses will not loop back to the starting point again and again. If you wait until the end to remedy things, the cost is often already buried in every earlier choice; this new path did not eliminate cost, but made cost appear earlier, making it easier to handle while changes were still possible.

Reveal

This story is about: KV cache

KV Cache stores the Transformer attention keys and values computed for previous tokens. During autoregressive decoding, the model can attend to the cached history instead of recomputing keys and values for the entire context at every generation step. It is essential for practical LLM serving, but it consumes GPU memory and becomes a major constraint for long context, high concurrency, and multi-tenant Agent workloads.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • The Long-Table Meeting: how an LLM generates output token by token
  • The clerk rereads all statements: without KV cache, historical context is recomputed repeatedly
  • Two types of cards: attention keys and values
  • The next representative checks the card: the decode phase reuses historical KV
  • The side hall is packed: KV cache memory pressure
  • Isolate by venue: cache isolation for different requests, tenants, and sessions
  • Final insight: KV Cache trades VRAM for speed and is a core resource for LLM serving latency and concurrency capacity

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

Attentioncontext windowdecodingPagedAttentionmemory optimizationinference engine
京ICP备2026014272号