← Back to concept explainers

Concept Fable

Speculative decoding

Conclusion first:Speculative Decoding uses a smaller or faster draft model to propose multiple candidate tokens, then lets the larger target model verify them in parallel. Accepted tokens speed up generation without changing the target model’s output distribution when implemented correctly. It reduces decode latency when the draft model is cheap and accurate enough, but poor acceptance rates can erase the gain.

Speculative Decoding · Inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The chief justice had the fast scribe write the draft first

The magistrate's court had a backlog of judgments. The chief justice judged accurately but slowly; the fast clerk wrote quickly but often made small errors. In the past, every judgment waited for the chief justice to write it word by word, and the line outside the hall grew longer and longer.

The county magistrate wanted the fast clerk to write the verdict alone. After three days of trying, simple cases were fine, but complex cases were full of errors and omissions. If the chief justice revised everything afterward, it became even more tiring.

The senior advisor proposed a compromise. The fast scribe first drafted the next few lines along the case, and the chief justice did not wait for the full document—he reviewed as he went: correct lines were stamped with a small seal, wrong places were sent back for rewriting, and the work resumed from the error.

At first, the fast clerk guessed too far ahead, and one mistake ruined a large section. The secretary limited him to pre-writing only a few sentences at a time and had him focus on common formats: case cause, evidence, sentencing. The chief justice focused on verifying these drafts, which was much faster than writing from a blank page.

For simple cases, the fast scribe had several sentences accepted in a row and judged documents very quickly; for complex cases, the chief justice would interrupt early and personally review slowly. Speed came from batch approval when guesses were accurate, and safety came from the chief justice still holding the final judgment.

The clerk proposed a compromise: let the fast scribe write the draft first, and the chief justice only accept or reject it sentence by sentence. If the first few sentences all pass, they go directly into the judgment; once any sentence is unstable, all later draft text is returned, and the chief justice rewrites from there.

At first the fast scribe wrote too loosely, and the chief justice rejected more, so speed did not improve. Later the fast scribe only rushed to write routine sentences he was confident about, leaving complex turns to the chief justice. The queue finally shortened: easy phrases were laid out in advance, and the final confirmation still came from someone accurate. Speed did not replace accuracy; speed only probed a usable path first. Outside the hall, common people only felt that judgments arrived faster, without knowing there was an extra acceptance check inside. If a draft was accepted, time was saved; if a draft was unreliable, it was discarded immediately. Speed came from verifiable lead time.

The prefecture later understood that slow-but-accurate people don't necessarily have to work slowly alone. Let the fast hand scout ahead while the accurate hand verifies in parallel: if the guess is right, time is saved; if wrong, roll back. The key is that the fast hand must not overstep authority to finalize, and the accurate hand must not abandon segment-by-segment verification.

Reveal

This story is about: speculative decoding

Speculative Decoding uses a smaller or faster draft model to propose multiple candidate tokens, then lets the larger target model verify them in parallel. Accepted tokens speed up generation without changing the target model’s output distribution when implemented correctly. It reduces decode latency when the draft model is cheap and accurate enough, but poor acceptance rates can erase the gain.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Chief Justice: target model
  • Fast scribe: draft model or assistant model
  • Draft sentence: speculative candidate tokens
  • One stamp covers several lines: parallel verification and token acceptance
  • Some sentence in the middle is wrong: rejection and fallback to target model decoding
  • Draft hit rate: acceptance rate
  • Final insight: Speculative Decoding uses a cheap model to guess ahead, then a strong model verifies, reducing generation latency without sacrificing quality

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

draft modeltarget modelverificationlatency reduction

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号