Concept Fable

Prefill

Conclusion first:The prefill stage is the stage at the very beginning of inference when the model first processes all input tokens in parallel and builds the KV cache. It occurs before token-by-token decoding and usually significantly affects time to first token.

Prefill Phase · Inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The secretary reads the entire file first, so the first instruction is grounded.

Every morning the governor's office received thick case files, attached to old rulings, land deeds, and correspondence. The new governor, impatient with his aides for not putting pen to paper, ordered that whoever wrote the first comment would be rewarded.

As a result, some people drew conclusions from the first page alone, while the old rulings tucked in later often overturned the earlier judgment. The chief of staff explained that the silence before the first sentence was not laziness, but reading the entire file into his heart and organizing it into a draft he could consult at any time.

He stipulated that they should first read through the shared background and attachments together, and then draft the endorsement. Cases of the same type could reuse existing indexes; when materials were too long, they should also be carefully trimmed, and decisive evidence must not be discarded for the sake of speed.

The first-sentence instruction still requires waiting, but subsequent writing is fast and coherent. The cost of long context often occurs before speaking; the more complete the preparation, the smoother the word-by-word progress afterward.

Concept Anchor

This story is about: prefilling

The prefill stage is the stage at the very beginning of inference when the model first processes all input tokens in parallel and builds the KV cache. It occurs before token-by-token decoding and usually significantly affects time to first token.

The larger the long prompt, RAG retrieval context, system instructions, and Agent state, the more noticeable the prefill cost. Prefix caching, prompt compression, chunked prefill, and scheduling strategies can all reduce or smooth this overhead.

Story Mapping

Petition and attachments: input prompt, retrieved context, tool state and history
Reading the full scroll: processing input tokens during prefill
First instruction: first generated token
Index and draft: KV cache populated by prefill
Same policy read ahead: prefix/prompt caching
Cutting material risk: unsafe prompt compression or context pruning
Final insight: Prefill Phase is the input processing cost before the first token, and long-context applications must manage it explicitly

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

KV cachedecodingprompt processingfirst-token latency

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

In the prefill and decode phases, one takes in the question and computes full attention once, while the other moves forward one token at a time as it emits text