← Back to concept explainers

Concept Fable

KV Cache

Conclusion first:When a large model generates text, each old token produces a Key and Value that the attention mechanism can use. By caching these Keys/Values, subsequent generation of new tokens does not require recomputing the entire history.

Key-Value Cache · Serving optimization

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The storyteller's two hidden ledgers

The port inquiry pavilion originally ran on a set of familiar methods. The old clerk knew the location of every cabinet's documents, and rarely made mistakes when searching at the old pace; merchants were also willing to wait for him to check slowly.

Later the matter grew. The merchant repeatedly pressed about the whereabouts of the same batch of goods, his urging stretching from early morning to dusk. The newcomer thought this was a matter of quantity—just send more people and add more hours; the veteran, however, saw that many details had already crossed the boundary the old methods could bear.

The person in charge tried the first method: flipping through the entire cabinet of documents every time. For the first few days there was indeed visible improvement, the queue was a bit shorter, and outsiders felt this craft had finally caught up with the new situation.

But trouble quickly changed shape. Some people skipped inspections to rush progress, some forced unsuitable work into old processes, and the most skilled people were dragged down by trivial matters. Errors were not always visible on the surface and often only emerged after delivery. The old ledger could still explain where things came from, but it could not explain how to change for the next step.

The person in charge added more rules, requiring everyone to sign more, report more, and add another layer of protection. On paper it looked thorough, but on site it was slower; the craftsmen were busy proving they were not wrong, while the real cracks continued to widen in place. Onlookers only saw results fluctuating between good and bad, and people on site found it increasingly difficult to judge whether to save speed, quality, or responsibility first. Everyone felt they had done a little more, yet the whole thing did not become more stable because of it.

A silent old craftsman laid several failures on the table. He blamed no one, only had everyone look at the same detail: the new restriction had never entered daily practice, and only suddenly appeared before delivery. He laid the several reworks out in the order they occurred, and only then did everyone discover that mistakes were always amplified at the same corner.

So the workshop changed to a new rule: write the goods names and their corresponding destinations on cards kept at hand, and check the cards first for anything already asked about. At first no one was used to it, and everyone felt tied up; but after a few rounds, the rough shortcuts were blocked, and the truly reliable moves gradually remained.

When the same pressure came again, delivery did not become fancy, but it was clearly more reliable. The old clerk finally understood: if the correspondences checked earlier can be retained, later question answering saves a great deal of searching. If remediation waits until the end, the cost is often already buried in every earlier choice; this new path did not eliminate cost, but it made cost appear earlier, so it could be handled while changes were still possible.

Reveal

This story is about: KV Cache

When a large model generates text, each old token produces a Key and Value that the attention mechanism can use. By caching these Keys/Values, subsequent generation of new tokens does not require recomputing the entire history.

KV Cache can significantly speed up autoregressive generation, especially for long conversations and streaming output. But it consumes VRAM or memory; the longer the length, the higher the concurrency, and the more model layers and heads, the higher the cache cost.

Metaphor mapping

  • Two hidden ledgers: the Key and Value caches of historical tokens
  • New questions consult the hidden ledger: new tokens make attention queries against historical context
  • No need to reread the whole book: avoid recomputing historical tokens
  • The secret ledger fills the table: KV Cache occupies VRAM
  • One secret ledger per table: concurrent requests each need their own cache

Andy Ren Xu's judgment

KV Cache is one of the underlying reasons for the cost of long context. What looks like simply "storing more history" in product design can directly turn into a bill for VRAM, latency, and concurrency capacity.

attentiondecodePagedAttention

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号