← Back to concept explainers

Concept Fable

Cache offloading

Conclusion first:KV cache offloading moves less active KV cache blocks from GPU memory to CPU memory or SSD. This allows the system to support longer contexts or more concurrent requests without being entirely limited by GPU memory.

KV Cache Offloading · Inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

After the council chamber moved the old files to the neighboring storage room

The port arbitration court handles long-contract disputes. On the judge's desk must be placed the contract, attachments, correspondence, and prior rulings. A short case fits on one desk, but a long case piles the desktop full.

As cases multiplied, the scribe began cramming all the files onto the main desk. As a result, new cases could not get in, old cases could not be turned, and the most expensive spot on the desk was occupied by old pages that were rarely consulted.

The director first bought a larger table, then told the clerk to write smaller. But long contracts grew even faster, the tabletop was soon filled again, and there was no place to find the passage actually under review.

Someone proposed throwing away the old files and keeping only summaries. After several misjudgments, everyone realized that some obscure clauses could suddenly determine the outcome, so they could not simply be discarded.

Later the archive keeper layered the files: pages under review on the main desk, ones possibly needed soon on the shelf behind, ones consulted only rarely in the next room. They could be retrieved when needed, but one had to accept it would be a little slower.

He also assigned a number to each case, ensuring that Company A's contract would not be pulled by Company B; there were also clear rules about which pages could leave the desk and which pages had to stay at hand.

The arbitration court could finally handle more long cases at once. The cost was that some lookups would be half a beat slower, but that was far better than letting the main table be crushed by all the files.

The director understood that the problem was not whether the files had value, but that different files occupied positions of different value. The most expensive desktop should be reserved for the hottest content that needs immediate access. From then on, long-contract cases no longer blindly competed for the main desk. What could be moved away was moved away; what had to be looked up quickly stayed. The system traded tiered storage for longer context and higher concurrency. The clerk also learned to record the cost of moving: move too much and it slows down; move too little and the main desk gets crushed. Every case found a balance between fast access and distant storage, rather than piling everything in front of them. The director later wrote this into the office rules: a precious position is not for the thickest file, but for the page most likely to be turned to next. In this way, slower retrieval bought more cases being heard at the same time.

Reveal

This story is about: cache offloading

KV cache offloading moves less active KV cache blocks from GPU memory to CPU memory or SSD. This allows the system to support longer contexts or more concurrent requests without being entirely limited by GPU memory.

The cost is that retrieving these caches is slower, so it's a trade-off between capacity and latency. A good strategy must decide which cache blocks stay "hot" in the GPU and which can be temporarily moved out.

Metaphor mapping

  • Main table: GPU memory
  • Next-door storeroom: CPU RAM or SSD
  • Long contract files: the large amount of KV cache generated by long context
  • Buying a bigger table: simply adding GPU memory
  • Throwing away old files: unsafe context truncation
  • Tiered storage: KV cache offloading policy
  • Half a beat slower: the access latency caused by offload
  • Final insight: KV Cache Offloading uses slower storage to expand capacity, suitable for long context and high concurrency, but the latency cost must be managed

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

KV cachelong contextmemory optimization
京ICP备2026014272号