Concept Contrast Fable

In the prefill and decode phases, one takes in the question and computes full attention once, while the other moves forward one token at a time as it emits text

Conclusion first:The prefill stage processes all tokens of the input sequence at once, computes their attention in parallel, and produces the generation state for the first token. The decode stage then autoregressively generates each subsequent token one by one, computing only the attention of the newly added token over the history each time. The two stages combine into one complete inference call, but their computational characteristics and bottlenecks are completely different.

The prefill stage processes all tokens of the input sequence at once, computes their attention in parallel, and produces the generation state for the first token. The decode stage then autoregressively generates each subsequent token one by one, computing only the attention of the newly added token over the history each time. The two stages combine into one complete inference call, but their computational characteristics and bottlenecks are completely different.

Fable

There were two bends on the post road. At the first bend, carts and horses entered the mountains together; at the second bend, they crossed the suspension bridge one by one

In the north there was a post road from the capital to the frontier, eight hundred li in total, and the two most time-consuming sections were in the middle. The first was called Dragon Gate Bay, where the procession suddenly entered a canyon flanked by two mountains. All carts and horses advanced abreast at the mouth of the valley; the road ahead was wide enough for ten riders side by side—this section was fast and fierce, pushing the whole group through the canyon in one breath. The second was called Single-Line Bridge; after leaving the canyon there was a narrow wooden suspension bridge that could only take one rider at a time. One rider after another, no parallel passage.

Old Qin, the dispatcher of the post road, had managed this route for fifteen years. He never calculated time by total mileage; instead, he split Longmen Bay and Yixian Bridge and calculated them separately. The time through Longmen Bay depended only on the team's total weight and formation width—the wider the formation and the more horsepower, the faster they passed. The time over Yixian Bridge depended on total length divided by each rider's speed—unrelated to formation, but related to weather, wind speed, and each rider's braking reaction.

One year the court allocated a speed-up budget to the post road. Old Qin's first plan put everything on Longmen Bay: widen the valley mouth, pave hard ground, and buy ten more fast horses. After all three projects were completed, Longmen Bay became twice as fast. But the First-Line Bridge was untouched. Overall annual travel time fell by only ten percent—Longmen Bay was originally the segment accounting for less than twenty percent of the full journey. Good horses and bad horses could show their difference at the valley mouth, but at the bridgehead they all queued.

The following year they changed the plan and split the budget in two. The first portion went to repaving the road at Longmen Bay; the second went to adding rails at First-Line Bridge and replacing the wooden suspension bridge with a small twin-arch bridge for two-way traffic. Both projects advanced in parallel, cutting total time for the year by forty percent. At year-end, Old Qin told the new post station manager over the accounts: Longmen Bay is a parallel batch-processing stage; First-Line Bridge is a serial one-at-a-time stage. Speed gains have to be viewed in two segments—if you only pour money into one end, everything left over is waste.

Old Qin wrote this experience into the courier route manual: first there is a wide road that lets the whole team advance side by side, then there is a narrow bridge that only one rider at a time can cross. The wide road depends on total weight, formation, and horsepower; the narrow bridge depends on the gap between each rider's hoofbeats. The first section pursues pushing more people through at once, and the second pursues keeping every step unbroken.

Lao Qin later hung a route chart at every post station along the line, marking Longmen Bay as "parallel · throughput priority" and First-Line Bridge as "serial · latency priority". He told the couriers: whichever segment the line is stuck at, use that segment's strategy to tune it. Don't use the method for fixing the valley mouth to fix the bridge; the bridge doesn't recognize horsepower.

Later a new dispatcher took over and complained to Old Qin that the front-line bridge was still too slow. Old Qin opened the records: all the budget you gave the bridge last year only replaced the indicator lights when each rider got on the bridge—that was what Longmen Bay should have done; only parallel segments need horses prepared in advance. What the serial segment really needed was to minimize the gap between crossings so the bridge deck would not rebound when each horse landed.

Prefill phase

The first stage of inference: all input tokens enter the model at once, and attention and intermediate states are computed in parallel. It is compute-intensive and throughput-first, with the bottleneck in compute power. It outputs the hidden state of the first generated token.

Decoding stage

The second stage of inference, generating each new token autoregressively one by one. Each time, only the new token's attention to history is computed. Memory-access intensive and latency sensitive, with the bottleneck in reading and writing the KV cache.

Story Mapping

Longmen Bay valley mouth: prefill—input processed in parallel all at once, throughput-oriented
Single-plank suspension bridge: decode—serial token-by-token generation, latency-oriented
Old Qin calculated time in two segments: prefill and decode have separate computational characteristics and bottlenecks
In the first year, invest only in Longmen Bay: optimize prefill but decode drags it down—overall latency improvement is limited
In the second year, invest in both segments: prefill and decode must be optimized separately to achieve overall speedup
Route map showing parallel and serial: in system monitoring, the prefill and decode metric panels are separate
The waste of indicator lights: using prefill optimization methods to optimize decode is completely the wrong approach

Landing on inference performance optimization

When troubleshooting inference latency, first break it down to see whether the bottleneck is in prefill or decode. Slow prefill shows up as high first-token latency—the longer the input, the slower it gets—and the optimization directions are batching, parallel computation and memory bandwidth. Slow decode shows up as long intervals between each subsequent token—the optimization directions are KV cache layout, memory access patterns and speculative decoding. When buying GPUs, if the business is mainly long input and short output (such as translation or summarization), prefill compute matters more; if it is mainly long output (such as code generation or writing), decode memory bandwidth and cache architecture should be examined first. Both stages run on the same hardware, but when paying for cards you need to see clearly which stage is eating the money.

Prefill Decoding stage