← Back to concept explainers

Concept Fable

Sampling

Conclusion first:Sampling is a decoding method that randomly selects tokens according to the model's probability distribution. It can increase diversity and reduce the mechanical feel of greedy output, but it also brings non-reproducibility and low-probability errors. When enterprise agents use sampling, they need to combine temperature, top-k, top-p, banned words, structured output constraints, and evaluation, and differentiate scenarios such as customer service, marketing, brainstorming, code, and compliance statements by risk.

Sampling · Decoding

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

Draw one usable ticket from the candidate basket

The seaside workshop town wrote replies for merchants every day. The old master craftsman used to have apprentices write the most standard sentence, and the replies were steady, but they looked like they were stamped from the same wooden seal.

Customers gradually grew dissatisfied. People who had lost goods received cold boilerplate, and people changing addresses received the same opening. The service was not wrong, but you could not hear that anyone had actually looked at the letter.

The apprentices' first attempt at patching was completely free-form. The letters came alive, but some promised compensation they couldn't give, and some overstated the shopkeeper's rules, leaving the front hall to clean up the mess every day.

The old craftsman brings out the candidate basket. At each next sentence, several optional slips are placed in the basket: more for the safest options, one or two for rare but suitable ones; dangerous slips are cleared out first.

The shopkeeper also adjusts how tightly the paper is drawn. Compensation, deadlines, and responsibility attribution use the most stable slips of paper; greetings, explanations, and reassurance can retain some variation. Low-scoring slips do not get into the basket just because they are novel.

Later, the replies were neither rigid nor erratic. The workshop understood that change must be tamed by boundaries. Which sentence gets drawn may look like chance, but behind it you must first decide which possibilities are even eligible to enter the basket.

On the eve of a festival, incoming letters suddenly doubled. If every letter were written identically, customers would feel brushed off; if every letter were handwritten freely, the shopkeeper feared promises would exceed the counter's authority. The candidate basket proved useful at this point.

The apprentices learned to first establish the immutable skeleton, then choose among different phrasings within a safe range. What they extracted was not arbitrariness, but the selection of one path among many acceptable ones. The workshop's replies began to feel human, yet still like a shop that follows the rules. Later, when newcomers started training, the old master no longer began with abstract terms. Instead, he had them review that earlier failure: where intuition had been relied on, why the first fix was insufficient, and how the new arrangement kept risk in a place that could be checked. Only when they could explain these details clearly, and point out when to keep using the approach and when to upgrade it, were they considered to truly understand the method. Later, they also used this method to drill newcomers: first have the newcomer do it the old way and see how errors arise; then redo it under the new rules and see which step was blocked in advance and which step left evidence. If a newcomer could only recite names, they would still misuse it in similar scenarios; only those who could state the applicable boundaries, the costs, and the remedies after failure were allowed to take on work independently.

Reveal

This story is about: sampling

Sampling is a decoding method that randomly selects tokens according to the model's probability distribution. It can increase diversity and reduce the mechanical feel of greedy output, but it also brings non-reproducibility and low-probability errors. When enterprise agents use sampling, they need to combine temperature, top-k, top-p, banned words, structured output constraints, and evaluation, and differentiate scenarios such as customer service, marketing, brainstorming, code, and compliance statements by risk.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Candidate basket: next-token probability distribution
  • High-scoring tickets are drawn more often: high-probability tokens are more likely to be sampled
  • Remove prohibited tickets: security and format constraints
  • Limited sampling: adding diversity within a controllable range
  • Two brush techniques: different business risks correspond to different decoding strategies

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

temperaturetop-ktop-pstochastic generationdecoding
京ICP备2026014272号