Concept Contrast Fable

KV Cache vs. prefix caching: one is the inference engine remembering every step, the other is the serving layer reusing someone else's memory

Conclusion first:Both terms talk about "caching," but what is cached and where it is cached are completely different. KV Cache is when the model, during token-by-token generation, stores the intermediate results of attention computation to avoid recomputing historical tokens — this is at the inference engine level. Prefix caching stores the results of prompt prefixes shared across requests at the serving layer, so multiple requests can reuse them — this is at the system scheduling level. Optimize the wrong layer, and the money is spent outside the GPU while the speedup never arrives.

Both terms talk about "caching," but what is cached and where it is cached are completely different. KV Cache is when the model, during token-by-token generation, stores the intermediate results of attention computation to avoid recomputing historical tokens — this is at the inference engine level. Prefix caching stores the results of prompt prefixes shared across requests at the serving layer, so multiple requests can reuse them — this is at the system scheduling level. Optimize the wrong layer, and the money is spent outside the GPU while the speedup never arrives.

Fable

Two kinds of offcut reuse in a shipyard

There was a shipyard by the sea, and every ship required laying a keel. The keel was the most material- and time-consuming component. Each time the old craftsman laid a keel, he had to recalculate the forces, draw lines, cut, and polish according to the ship type. To build ten ships, he had to start from scratch each time.

The factory made two improvements one after another. The first improvement was having craftsmen record already-calculated keel data in a notebook—which wood was used for the keel at which rib position on each ship, and what the load-bearing data was. When building the next ship of the same type, the craftsman could pull out the notes and reuse them directly without recalculating the entire keel. This notebook stayed in the craftsman's toolbox and was used once for each ship built.

The second improvement was the factory manager's idea. He noticed that many ships' bow sections were designed exactly the same—no matter how long the hull or how much load, the bow curvature template was uniform. He set aside a wall in the design room, covered with standard processing drawings for bow sections, so any craftsman could take them directly. Different ship types, different orders—as long as the bow section was the same, it could be referenced directly.

These two kinds of reuse are often treated as the same thing, until a tight deadline causes a problem. The factory manager urges the craftsman to speed up, and the craftsman says: I can't find the first half of the data for this hull type in my notebook, so I have to calculate from scratch. The factory manager points to the standard bow diagram on the wall and says: Isn't this section posted on the wall? The craftsman explains: What's on the wall is the standard machining template for the bow; what's in my notebook is the mechanical calculation process for the keel from bow to stern—the bow section does have a template, but starting one-third of the way back from the bow, the data for every frame is different.

Later, the factory established a clear division of labor. The craftsman's notebook was responsible for reusing computation for the same ship—full-length, position-by-position, indivisible continuous computation. The standard drawings on the workshop wall were responsible for reusing common segments across different ships—only looking at prefix segments, shared across ships, used once then replaced by the next. The two ran in parallel, but neither should pretend to replace the other.

Once they took a large order and needed to build four ships of different specifications at the same time. Notebook reuse reduced the keel calculation for each ship by sixty percent, and wall reuse nearly eliminated manual work on the bow sections of all four ships. The factory director later included a separate budget line for walls and notebooks in the expansion plan.

At the year-end review, the technician drew a diagram: the notebook corresponds to the KV Cache inside the inference engine—avoiding repeated computation within the generated sequence; the wall template corresponds to the service-layer prefix cache—reusing a common prefix across requests. Both lines use the same batch of wood, but the data is managed by two different people and relies on two different lookup logics.

KV Cache

During autoregressive generation, the Transformer model stores already-computed attention key-value pairs to avoid recomputing attention over the historical sequence for each new token.

Prefix caching

At the inference service layer, cache the KV Cache of prompt prefixes shared by multiple requests in memory or on disk, so different requests reuse the computation results of the same prefix.

Story Mapping

Craftsman’s notebook records the full calculation data for the keel: KV Cache, reusing per-position computation within a sequence
Standard ship-bow template on the workshop wall: prefix caching, sharing the computation of common prefix segments across requests
The notebook cannot flip to the earlier section but the wall has it: two kinds of caching act on different scopes
Four ships of different specifications built at the same time: two caches work in parallel to reduce computation at different layers

Applying it to an Agent project

When evaluating inference acceleration options, first ask which layer is being optimized. KV Cache optimization relies on inference engine parameter tuning and VRAM management; prefix caching optimization relies on serving-layer scheduling strategy—the reuse logic for system prompts and shared context. Review budgets for the two directions independently; don't mistake the prefix cache's acceleration rate for the inference engine itself getting faster. One manages continuous state in VRAM, the other manages the shared starting point across requests.

KV Cache Prefix caching