← Back to concept explainers

Concept Fable

Chunked Prefill

Conclusion first:Chunked Prefill splits the prefill processing of long prompts into smaller chunks so a long request does not monopolize the GPU scheduler. This helps interleave long-context requests with shorter decode work, improving responsiveness and fairness while still eventually processing the full prompt.

Chunked Prefill · Serving optimization

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

When a long scroll enters the hall, it no longer covers the entire table at once

The archive received a very long scroll of maritime trade contracts. The old rule required first spreading out the entire scroll, numbering it and reading it through, and only afterward allowing anyone to process the next small application. That day the long scroll occupied the only large table, and even a small note asking only for an address had to wait. The archivists were not slacking off, but short matters were blocked outside by long matters, and the queue grew more and more anxious. The director first ruled that the long scroll could only come in the morning. The afternoon did become easier, but the morning turned into a disaster; and real business does not arrive according to the director's timetable.

Someone suggested returning the long scroll and asking the client to shorten it. The client said the contract was long by nature and key clauses could not be deleted just because the hall's table was small.

Later the dispatcher split the long scroll pre-reading into several segments. The librarian processed one segment first, freed the desk, then inserted a few short notes, and then continued with the next segment of the long scroll.

The long scroll is not discarded; it will still be read in full in the end. The difference is that the pre-reading work no longer swallows all compute slots at once, and short requests can enter between segments.

The bureau began recording the size of each segment, the switching cost, and the wait time for short slips. Cut too fine and paper gets moved back and forth; cut too coarse and it blocks others.

The curator finally saw that the trouble with long inputs was not just their total size, but that the prefill phase easily monopolized resources. Scheduling prefill in chunks allows long and short requests to coexist in the same serving system.

The dispatcher cut too finely the first time—the librarian spent half a day just rolling up, putting down, and re-finding positions. The second time the cuts were too coarse, and the short slips got blocked again. Only then did the director treat segment length as a craft to be tuned, not a casual snip.

Later, long contracts, short addresses and medium bills all arrived at the library at the same time. The big table seemed to breathe, long scrolls advanced steadily, and short matters no longer starved at the door. Everyone finally understood: fairness is not making every piece of paper the same length, but preventing resources from being monopolized by any one piece of paper. This craft let the librarians respect the integrity of long scrolls while also accommodating the waiting cost of short slips. Long and short work in the library could finally advance in turn on the same table.

Reveal

This story is about: Chunked Prefill

Chunked Prefill splits the prefill processing of long prompts into smaller chunks so a long request does not monopolize the GPU scheduler. This helps interleave long-context requests with shorter decode work, improving responsiveness and fairness while still eventually processing the full prompt.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Extremely long maritime trade contract: long prompt / long context request
  • Unroll the whole scroll before handling others: monolithic prefill blocking the scheduler
  • Split into several segments for pre-reading: chunked prefill
  • Short notes inserted: short requests interleaved between chunks
  • Segment size and switching cost: scheduling tradeoff
  • Read in full: chunking changes scheduling, not the semantic input

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

prefillscheduling
京ICP备2026014272号