Concept Contrast Fable

In the prefill and decode phases, a whole batch of input is swallowed in one gulp, then spat out one character at a time

Conclusion first:Prefill is the first stage of inference, processing all input tokens in parallel at once. The decode stage follows immediately, generating output tokens autoregressively one by one. Some people casually refer to prefill as the 'input processing' of inference, but decode is what actually produces the results users see—the engineering characteristics of the two stages are completely different.

Prefill is the first stage of inference, processing all input tokens in parallel at once. The decode stage follows immediately, generating output tokens autoregressively one by one. Some people casually refer to prefill as the 'input processing' of inference, but decode is what actually produces the results users see—the engineering characteristics of the two stages are completely different.

Fable

One stove with two pots, one over high heat stir-frying the whole batch, one over low heat simmering and serving spoon by spoon

In the town there was a restaurant specializing in official banquets, and the main kitchen was run by two brothers. The older brother managed the wok: ingredients went into the pan together, stir-fried quickly over high heat, and plated in two or three minutes—prep had to be complete, and if one ingredient was missing, the stove was not lit. The younger brother managed the soup pot: the broth was simmered in the morning, and when guests ordered, he ladled it into bowls one spoon at a time, checking the oil on the surface and the ingredients at the bottom of the bowl with each spoonful, never rushing.

The efficiency of a restaurant lies entirely in the coordination in the kitchen. When the wok cooks fast, one dish after another is pushed out, and dishes for ten tables come out at the same time. But the soup pot is slow—each bowl must be ladled, checked and seasoned separately. Between the wok and the soup pot there is a serving counter, where cooked dishes pile up waiting, and the bottleneck is always on the soup pot side.

For a while the restaurant took on many large birthday banquets—single tables with enormous quantities of food. The wok's ingredient load jumped from one person's portion to a whole table's portion; the wok was still the same wok, the heat unchanged, but the time to toss one batch nearly tripled. The soup pot's per-bowl speed didn't change, but overall waits grew longer. The kitchen manager went through the complaints one by one and found that seventy percent were not about slow serving but about "too big a gap between dishes and soup"—the stir-fried dishes arrived, but the soup still took a long time.

The supervisor did two things. The first was to strengthen the wok station's prep standards—no matter how large the order, prep must be completed within two minutes, and there must be no idle waiting where the wok is on the fire but one ingredient is missing. The second was to add a heat-retaining portioning station to the soup pot—the soup pot's main broth is first pre-portioned into ten bowls at the station, so each customer order only needs final seasoning and garnish, speeding up the slowest end.

After the correction, the negative reviews of the birthday banquet disappeared. The head of the back kitchen told the new apprentice: the wok processes the entire input in parallel at once—it depends on computational density, and the larger the amount of food, the stricter the timing requirements for all ingredients in the wok. The soup pot outputs one by one—it depends on single-step smoothness, and the way to speed it up is to reduce the preparation time between steps. One kitchen has two pots, but when you put fresh vegetables into the wok, the soup pot is still moving; they do not pause each other.

The supervisor wrote this experience into the kitchen handbook: the wok depends on prepped ingredients being ready in one go and high-heat stir-frying, while the soup pot depends on no gaps between each ladle. If the wok is slow, check the prep and the heat; if the soup pot is slow, check the plating station and the soup-serving rhythm. The two pots get stuck in different places, so the fixes differ too.

Later someone asked the supervisor whether the wok and the soup pot could be merged into one pot. The supervisor thought for a moment and said the wok uses high heat for a short time, while the soup pot uses low heat for a long time—putting them on the same medium would only make both worse. What you can do is let the hot dish from the wok slide directly into the serving trough beside the soup pot, without cooling and reloading in between. The boundary does not move, but the flow must be smooth.

Prefill

The first stage of inference. The entire input sequence of tokens is processed in parallel at once, computing attention for all tokens and outputting the generation state of the first token. Throughput-oriented and compute-intensive.

Decoding stage

The second stage of inference. Output tokens are generated autoregressively one by one, and at each step attention over the history is computed for the new token. Latency-sensitive and memory-access-intensive.

Story Mapping

Wok (fierce high-heat stir-fry): prefill—parallel attention computation over all input tokens
Soup pot (simmer over low heat, ladle out spoon by spoon): decode—autoregressive generation token by token
Dish-passing counter congestion: prefill is fast but decode is slow, causing perceived user latency
A large birthday banquet wok takes three times as long: prefill latency growth for long inputs
Strengthening the wok prep standard: the key to prefill optimization is reducing preparation time differences
Soup pot insulation and portioning station: decode optimization relies on reducing the waiting interval between single steps
Discussion of merging the two pots: the computational characteristics of prefill and decode cannot be merged into one kernel
KV Cache Reuse: the intermediate state output by prefill is passed directly to decode without a cold restart

Falls into latency analysis for inference services

When users feel "the model is slow to respond", first look at two separate metrics: first-token latency and subsequent token interval. High first-token latency means prefill is the bottleneck—the input is too long or the batch too large, and the prefill parallelism strategy needs optimizing. A large subsequent token interval means decode is the bottleneck—single-step memory access is slow, and KV cache reads/writes need optimizing or speculative decoding should be tried. On the service monitoring dashboard, set up separate panels for TTFT (time to first token) and TPS (tokens per second), because they correspond to two different optimization paths. If you optimize only prefill and are satisfied with latency, long-output scenarios after launch will expose decode's hidden weakness.

Prefill Decoding stage