← Back to concept explainers

Concept Fable

Decoding stage

Conclusion first:The decoding phase is the autoregressive generation phase after prefill: the model generates one token at a time, reads the KV cache at each step, computes the next token, appends the new cache, and may stream tokens back to the user.

Decode Phase · LLM inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The lighthouse only turns out the next beam each time

The strait lighthouse sends signals to ships at night. The keeper first reads the wind direction, tide, and ship positions, then begins turning the lamp mirror in sequence. After the first beam lights up, the crew feels somewhat reassured, but the complete signal is still long. Each beam must follow the previous one, or ships far away will read the wrong meaning.

The mayor first asked the lighthouse keeper to shine the entire signal at once. The keeper shook his head: the position of the lamp mirror, the direction of the previous beam, and the choice of the next beam affect one another and cannot all be sent simultaneously.

Someone had multiple apprentices each take a section, and the result was broken signals, and the ship misunderstood the port instructions. Work generated sequentially cannot simply be split among multiple people in parallel.

The old lighthouse keeper improved things elsewhere. He recorded the wind directions and ship positions already used next to the turntable, updating them with each beam of light, no longer re-checking the entire logbook. He also had multiple lighthouses work on the same rhythm: each lighthouse sent its own next beam, sharing fuel and tempo, but without disrupting their individual sequences.

Strait throughput became higher, and waiting per ship became more controllable. What truly limited speed was the dependency between each beam of light that could not be skipped, and the computational cost of each round's cadence. The mayor understood that guarding the tower meant sending signals one beam after another. Each beam had to look at the previous direction before deciding how the next beam should turn.

The young lighthouse keeper thought it was too slow and wanted to set all the lamp mirrors at once. The old keeper stopped him: the previous beam changes the meaning of the next; only after a ship sees 'north' does the following 'three quarters' mean three quarters of an hour heading north; if the order is wrong, the whole string of signals changes meaning.

Later the sea fog thickened, and the lighthouse keeper had even less room for error. With each beam he sent, he waited for the gears to reset, then adjusted the next beam based on the light just sent. Sometimes a steady choice early on made everything smooth later; sometimes hesitation early on forced later changes too. What the ships received was not a blob of light, but sentences growing one beam after another.

Later, when optimizing the lighthouse, they no longer focused only on the first beam. The steady rhythm after the first beam is what determines whether a long signal can withstand storms and night voyages.

Reveal

This story is about: the decoding stage

The decoding phase is the autoregressive generation phase after prefill: the model generates one token at a time, reads the KV cache at each step, computes the next token, appends the new cache, and may stream tokens back to the user.

Autoregressive dependency means the next token must be determined based on the previous token; the KV cache is a cache of prior computation results. The decoding phase directly affects tokens per second and total completion latency, especially for long answers and real-time agent experiences.

Metaphor mapping

  • Lighthouse signal: streamed model output
  • After the first beam of light, you still have to continue: TTFT is only the start; decode controls remaining latency
  • Each beam of light in order: autoregressive dependency
  • Record beside the turntable: KV cache
  • Multiple lighthouses beating in the same cycle: batching multiple sequences per decode step
  • Fuel and cadence: GPU utilization and scheduler cadence
  • Final insight: the core of the Decode Phase is the per-token hot path, which determines output speed and the long-answer experience

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

prefillKV cache
京ICP备2026014272号