← Back to concept explainers

Concept Fable

Paged attention

Conclusion first:PagedAttention is the specific vLLM-style implementation of paged KV cache management. It uses block tables to map a sequence’s logical KV positions to physical cache blocks, reducing memory waste and enabling efficient batching for many concurrent requests. In production LLM serving, it is a core technique for improving throughput under long contexts and mixed request lengths.

PagedAttention · LLM inference optimization

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

After the shipping accountant changed the cargo hold into numbered compartments

On the river there was a large cargo ship that often took on goods of varying lengths. The old shipping method reserved a full row of cabin space for each passenger, regardless of whether they only placed three boxes or thirty. Short cargo occupied empty space, and long cargo often couldn't fit, so the shipowner always complained the ship was too small.

The new hold manager cut the ship's hold into many numbered small compartments. A passenger's goods didn't need to be stored together; they just had to write clearly on the cargo ticket the first compartment, the next compartment, and the order. Three boxes of goods took three compartments, thirty boxes took thirty compartments, and empty compartments could be given to others at any time.

The sailors were initially afraid of finding disorder. The hold keeper made a mapping tag for each shipment: which box was in which slot, which slot the next box connected to, and if a slot was missing it could be checked immediately. Cargo no longer lined up by memory, but kept its order by numbering and tags.

Once a big customer added ten boxes at the last minute. The old method required shifting an entire row of cargo space. The new method only had to find scattered empty slots to connect, and a few lines on the mapping board were enough. After another customer with a short shipment disembarked, the vacated slots could immediately be used by new cargo.

The ship did not get bigger, yet it could take more passengers. Long cargo no longer forced everyone to continuously yield space to it, and short cargo no longer wasted a whole row of space. The cost is that cabin management must be precise; if a tag is off by one slot, the cargo order will be disrupted.

The new method initially made the sailors uneasy: with goods scattered across compartments, would things get lost? The hold keeper spot-checked the mapping tags every day and found that as long as the tags were reliable, goods did not need to be crammed into consecutive slots. Empty spaces were reused, and both big and small clients waited less.

Once, as a storm approached, the shipowner needed to free up a section of hold space for emergency grain. The old ship method could only shift cargo in whole rows, but the new grid could assemble scattered empty spaces according to the mapping tags. The cargo order was not disturbed, and no space was wasted. Only later did the shipowner dare to accept more scattered orders. Short cargo no longer wasted whole rows of hold space, and long cargo could be distributed as well. The numbered grid and mapping tags let space flow like living water, while the cargo order could still be recovered.

The shipowner later understood that high turnover does not necessarily come from a bigger ship, but also from cutting the underlying space into trackable, reusable small blocks. As long as order and boundaries are governed by signs, scattered space can also serve many requests of different lengths.

Reveal

This story is about: paged attention

PagedAttention is the specific vLLM-style implementation of paged KV cache management. It uses block tables to map a sequence’s logical KV positions to physical cache blocks, reducing memory waste and enabling efficient batching for many concurrent requests. In production LLM serving, it is a core technique for improving throughput under long contexts and mixed request lengths.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Cargo hold: GPU memory used by KV cache
  • Contiguous slots: naive per-request contiguous allocation
  • Fixed numbered slots: PagedAttention blocks
  • Ledger mapping table: logical-to-physical block table
  • Mixed loading of long and short goods: mixed sequence lengths in serving
  • More customers on the same boat: higher concurrency and throughput
  • Final insight: PagedAttention manages the KV cache as mappable blocks, letting serving systems waste less VRAM and batch more effectively

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

vLLMKV cachebatchinginference throughputGPU memory
京ICP备2026014272号