← Back to concept explainers

Concept Fable

Speculative sampling

Conclusion first:Speculative sampling, often also described as a form of speculative decoding, first samples candidate tokens through a draft process, then has the target model accept or reject them. The final output should still preserve the target model's original distribution.

Speculative Sampling · Inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The wine taster pours three cups first; the winery owner only confirms whether it can go into the barrel

The print shop was rushing an urgent letter, and the old proofreader had a good eye but was slow. The young apprentice was quick and could often guess how the next line should be typeset, but occasionally made outrageous mistakes.

The manager first had the young apprentice typeset alone. Speed went up, but so did typos. Then he used only the old gentleman, and the letters could not keep up with the post horses. Each side had its shortcomings.

Later the print shop switched to a relay. The young apprentice first drafted several lines of candidates in one go, and the old master followed right behind; correct lines were stamped and passed, and wrong ones were sent back from the point of error for the old master to redo himself.

At first the young apprentice was too fond of taking risks, guessed wrong repeatedly, and actually slowed things down. So the proprietor limited the length of each guess and picked the letters he was good at by subject. The old gentleman still kept the final judgment.

Later, the shopkeeper assigned topics to the young apprentice: for familiar family letters, he could guess a few more lines; for official contracts and prescriptions, he could only guess briefly. The old gentleman also stopped rereading every correct line word by word, and only paused where the young apprentice might drift off course.

The urgent letter finally caught up with the post horse. The print shop did not replace Slow Mr., nor did it indulge the fast apprentice to run wild; it let the fast side scout ahead and the accurate side check at the gate. A correct guess saved time, and a wrong guess was promptly withdrawn.

The bookshop also tracked which sentence patterns the young apprentice often guessed correctly. With fixed greetings and common sign-offs, he could run ahead; with numbers, medicine names, and contract terms, the old gentleman immediately took back the pen. The value of speed comes from verifiability, not from blindly rushing ahead. Later the old gentleman discovered that he was not slowly rearranging every line, but confirming at key points. If the young apprentice probed accurately, the whole passage went fast; if he probed off target, the retreat was also timely. Speed came from this retractable forward probing. The shopkeeper later wrote this relay into a rule: probe fast first, verify strictly afterward, withdraw immediately when wrong, and do not let recklessness roll into a major error. Only then was speed stable. The bookshop finally took care of both ends. Less waiting. Business more stable. Good reputation thus gradually took hold, and the courier station was willing to entrust them with urgent items.

The print shop became much faster as a result. The trick was not to let the fast hand replace the accurate hand, but to let the fast hand scout first and the accurate hand quickly verify; if it was right, time was saved, and if it was wrong, they returned to the reliable path.

Reveal

This story is about: speculative sampling

Speculative sampling, often also described as a form of speculative decoding, first samples candidate tokens through a draft process, then has the target model accept or reject them. The final output should still preserve the target model's original distribution.

It is suited to reducing decoding time when sampling generation is needed: the draft process takes a few trial steps first, and the target model only verifies the acceptable parts. The higher the acceptance rate, the more obvious the speedup.

Metaphor mapping

  • Village chief: target model distribution
  • Wine taster: draft sampler or small model
  • A few glasses of sample wine: sampled candidate tokens
  • Village chief continuously accepts: accepted speculative tokens
  • First deviation readjustment: rejection and correction
  • Do not only give the easiest choice: maintain sampling distribution correctness
  • Final insight: the core of Speculative Sampling is to accelerate sampling while making the final output still look like it was sampled by the target model itself

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

speculative decodingdraft modeltarget model
京ICP备2026014272号