← Back to concept explainers

Concept Fable

Top-k sampling

Conclusion first:Top-k Sampling limits the next token candidates to the k highest-probability options, then samples from among them. It can cut off long-tail low-quality tokens and reduce the risk of random sampling going out of control. The drawback is that k is fixed and does not consider the shape of the probability distribution: sometimes reasonable choices exceed k, and sometimes the top k still contains overly broad candidates. In enterprise use, it is usually tuned together with temperature, top-p, task type, and evaluation.

Top-k Sampling · Decoding

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

Only let the top ten craftsmen into the lottery box

The Lantern Riddle Shop gave guests a second line every night. The craftsmen first came up with many candidate lines, some elegant, some mediocre, and some novel but too far off topic. If they always chose the line that fit best, the riddles were very stable, but lacked surprise.

The workshop owner tried drawing lots from all candidates. The first night was lively, but on the second night an absurd second line was drawn, and the guests roared with laughter. With everything fully open, even unqualified lines got mixed in, and the liveliness quickly turned into chaos.

The old craftsman proposed setting a threshold first: each time, put only the most promising first few lines into the bamboo tube, then draw from the tube. This preserved a little variation while keeping the chaotic tail lines out. Hearing this, the apprentice thought that as long as the top ten were fixed, everything would be fine.

On the wedding day, ten lines were placed in the bamboo tube, and although the line drawn was acceptable, it was not solemn enough. On the poetry gathering day, only three lines were placed, and it felt too rigid. The workshop owner had them adjust by occasion: for weddings, prioritize stability, so only three to five lines go in the tube; for poetry gatherings, prioritize novelty, so ten lines can go in; for serious topics, better to put in fewer.

Later they discovered that this threshold had rough edges too. The tenth and eleventh places were sometimes nearly equal, yet were hard cut apart; sometimes the top three far outperformed the rest, making ten sentences redundant. The shopkeeper treated it as a simple tool, not an always-correct decree.

The first version of the draw also missed checking the theme. Some sentences ranked high but clashed with the tone of the preceding text. The old craftsman added a re-read: first sort candidates by strength into the bamboo tube, then decide how wide the opening should be based on the occasion and theme.

The riddle workshop eventually settled into a habit: first let candidates pass the qualification line, then preserve randomness within the qualification line. This avoided both rigid uniformity and wild divergence. The value of drawing lots among the top few was to draw a safe boundary around variation first.

Later, the lantern riddle workshop would occasionally narrow the bamboo tube so the steadiest sentences could go straight on stage; it would also relax a bit on garden night so novel sentences had a chance to emerge. The workshop owner knew that variation must be bounded, and the bounds must move with the occasion. Fixing the top few is only a tool, not aesthetics itself. Later, the apprentices understood that a bamboo tube mouth too wide or too narrow would ruin the mood of an evening.

Reveal

This story is about: Top-k sampling

Top-k Sampling limits the next token candidates to the k highest-probability options, then samples from among them. It can cut off long-tail low-quality tokens and reduce the risk of random sampling going out of control. The drawback is that k is fixed and does not consider the shape of the probability distribution: sometimes reasonable choices exceed k, and sometimes the top k still contains overly broad candidates. In enterprise use, it is usually tuned together with temperature, top-p, task type, and evaluation.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Long list: all possible tokens
  • Top k: the k candidates with the highest probability
  • Lottery box: sampling from the top-k set
  • Different k for different orders: controlling diversity and risk by scenario
  • Hard-cutting reasonable choices: the fixed-threshold limitation of top-k

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

samplingtop-ptemperaturedecoding

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号