← Back to concept explainers

Concept Fable

Autoregressive decoding

Conclusion first:Autoregressive decoding, also known as the Decode Phase, is the stage in which a large model generates tokens one by one after processing the input. Each new token depends on what has already been generated, so it naturally has sequential dependencies and cannot be fully parallelized like batch reading of input.

Autoregressive Decoding / Decode Phase · Inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

In a linked-poem gathering, every line must follow the previous line

Every evening the lakeside academy held a chain-poetry gathering. The host gave the first line, and each poet after that could only add the next line based on all the lines before. For short poems, this game was elegant.

Later, the prompts grew longer, with allusions, rhymes, guest requirements, and forbidden phrases placed alongside. With each line the poet had to grasp the prior context, growing slower and slower while more and more people waited.

The director first had the poet memorize allusions in advance and also swapped in better paper and brushes. But the rules of the chain poem did not change: the next line must wait for the previous line to appear; ten lines cannot be written simultaneously. Preparation can happen in advance, but putting pen to paper still has to queue.

Someone suggested writing the whole poem in one go. After a few attempts, the rhymes were off, the characters contradicted each other, and the ending could not connect to the beginning. Only then did everyone realize that this kind of poem must be continued line by line, and jumping ahead would only leave later lines without a source.

The recorder began preserving every clue that had already taken shape. When the poet continued with the next line, he no longer had to reorganize all the allusions, but only used these records and the latest line to keep judging. The academy did indeed become a little faster.

Even so, each step still had a fixed cost: fetching clues, thinking of the next word, writing it down, and writing the new clue back to the record. The longer the poem, the thicker the record, and the more poems arriving at the same time, the more crowded the academy became.

The director then viewed the poetry gathering in two phases: the opening reading of prompts could be prepared in parallel by many people, but the actual chain composition advanced line by line. The bottlenecks of the two phases differ and cannot be rushed with the same method.

Later, when the academy evaluated speed, it no longer only asked whether the topic was read quickly, but also specifically looked at how many characters could be chained per breath and whether there was idle waiting at each step. The root of slowness was hidden in the sequence; what could be changed was the way records, queuing and bystanders pre-testing were done. The academy also found that the further into the latter half of the poem, the more precious the clues left earlier became, and the more space they took up. If records were scattered, the poet would search back and forth repeatedly; if records were too sparse, key foreshadowing would be forgotten. The speed of the chain was often dragged down bit by bit by these small steps. Some customers urged urgently, but the dean only dared to prepare more candidate lines on the side, not dismantle the sequence. Because once the previous line changed, all subsequent expectations had to change with it. Later, during the retrospective, everyone wrote this lesson into the daily rules: first look at where on site the result was truly changed, then decide what to keep, what to repair and what to restrict.

Reveal

This story is about: autoregressive decoding

Autoregressive decoding, also known as the Decode Phase, is the stage in which a large model generates tokens one by one after processing the input. Each new token depends on what has already been generated, so it naturally has sequential dependencies and cannot be fully parallelized like batch reading of input.

KV Cache can avoid repeatedly recomputing historical context, but the next token still has to wait for the previous token to emerge. In serving performance, tokens per second, total generation latency, continuous batching and speculative decoding are largely optimized around this stage.

Metaphor mapping

  • Chain poem gathering: autoregressive token generation
  • The previous sentence determines the next: causal dependence between tokens
  • Opening question reading: prefill phase
  • Save the clues: KV cache
  • Each step takes a clue and writes it back: decode step using and updating cache
  • Cannot write ten sentences at once: the sequential nature of autoregressive decoding
  • Final insight: the Decode Phase is a per-token hot path, and optimization should focus on caching, scheduling, speculation, and per-step compute cost

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

prefillKV cachetoken generationlatency
京ICP备2026014272号