← Back to Concept Contrast

Concept Contrast Fable

The fast-guess, slow-verify method and the small model responsible for fast guessing

Conclusion first:Speculative decoding is an acceleration strategy in which a small model quickly generates candidates, which a large model then verifies in parallel. The draft model is the small model responsible for the quick guesses. The two concepts often appear together in one architecture diagram, but one is a method and the other is a component within the method—equating the method with the component will lose sight of the overall strategy when replacing or upgrading the draft model.

Speculative decoding is an acceleration strategy in which a small model quickly generates candidates, which a large model then verifies in parallel. The draft model is the small model responsible for the quick guesses. The two concepts often appear together in one architecture diagram, but one is a method and the other is a component within the method—equating the method with the component will lose sight of the overall strategy when replacing or upgrading the draft model.

Fable

The head chef had the apprentice chop vegetables while he only tasted—twice as fast, but the apprentice cannot be replaced

In the east of the city there was an old established restaurant. Chef Zhang cooked alone—slow, but he never made a mistake. Business grew better and better, with lines stretching to the street corner, and Zhang found his spatula could not keep up with the orders. He came up with an idea: let his apprentice Li Er first prepare the main and auxiliary ingredients for each dish, and he himself would only handle the final three minutes in the wok. When orders piled up, Li Er prepared the ingredients for four or five dishes at once on the cutting board, and Zhang cooked one after another, doubling the serving speed.

This method worked well for a year. Then the kitchen changed to a new menu and added three dishes requiring special knife skills—sliced fish needed thin diagonal cuts, deboned chicken needed joint seams opened, and carved garnish needed more than a dozen continuous cuts. Li Er's knife skills had only been trained on home-style dishes, and the new dishes came out in uneven pieces. When Chef Zhang put them in the wok, he found the heat was wrong: some were too thick and undercooked, some too thin and burnt at the bottom.

Master Zhang tried two directions. The first was to have Li Er retrain his knife skills—Li Er cut two hundred jin of ingredients every day, trained for two months, and barely passed three dishes on the new menu, but by then the kitchen had changed menus again. Master Zhang realized that the menu changed faster than knife skills could be trained. The second direction was to replace Li Er with Wang San—Wang San had worked five years as a prep cook at another large restaurant and had comprehensive knife skills. After the switch, the new menu no longer had problems.

The diners thought the kitchen method had changed, but actually the person had changed. Chef Zhang's method was unchanged—apprentices still cut ingredients, and the chef still checked the wok. But the efficiency of that method was directly tied to who was doing the cutting. An old regular once asked Chef Zhang why the kitchen had gotten so much faster, and Chef Zhang said: the method is called taste-first prep, but the one really doing the work is that prep cook. Many people lump methods and tools into one phrase, and when they take it to compare with other restaurants, there's no way to compare.

Wang San worked for half a year, and new problems emerged. After the hot dishes line launched, cold dishes and soup bases needed to be prepared simultaneously, and Wang San alone couldn't keep up. Master Zhang added another apprentice specifically for cutting cold dishes. Now the method was still taste-first-then-prepare, but the component went from one person to two. If the method had been hard-bound to one apprentice, adding a person would have required rewriting the entire architecture description.

The kitchen later separated the two things in the training manual: one page described the process—the methodology of apprentices preparing ingredients in advance and masters validating in parallel; another page described the people—who is currently responsible for ingredient preparation, what the skill scope is, and when to switch people. After the separation, changing the menu required updating only the people page, and changing the process required updating only the methodology page.

Before retiring, Master Zhang said one thing: discussing only methods is not enough to make the kitchen run fast, and discussing only people is not enough to explain what the kitchen is doing. Clarify both, and you need not fear the menu changing three times a year.

Speculative decoding

An inference acceleration method that uses a small model to quickly generate multiple candidate tokens, which are then verified in parallel by a large model. The core is speculative execution and the verification game, and it is not tied to any specific small model.

Draft model

The small model in speculative decoding responsible for quickly generating candidate sequences. The draft model can be replaced, upgraded, or combined—it is just a component in the speculative decoding strategy, not the strategy itself.

Story Mapping

  • Taste-first pairing method: speculative decoding, an overall acceleration strategy where a small model guesses quickly and a large model verifies
  • Li Er cuts vegetables: draft model, a small model responsible for quickly generating candidates
  • New menu error: when the draft model lacks capability, the overall throughput of speculative decoding drops
  • Swap Li Er for Wang San: the draft model can be replaced, but the speculative decoding method stays the same
  • Add a cold-dish apprentice: the draft model for speculative decoding can combine multiple components
  • The training manual has two pages: methods and components should be described separately in engineering documentation

Translate into inference acceleration solution design

When designing a speculative decoding scheme, first confirm which draft model suits your task type—a smaller version of the same model, a dedicated lightweight model, or an n-gram-based statistical draft. The draft model's acceptance rate directly affects the speedup ratio, so run an acceptance-rate benchmark before swapping the draft model. When a vendor says it uses speculative decoding but does not specify the draft model's exact composition, the speedup promise lacks binding force.

京ICP备2026014272号