← Back to concept explainers

Concept Fable

Best-of-N sampling

Conclusion first:Best-of-N sampling can also be understood as "selecting the best among multiple candidates": the system generates N candidate answers at once, then uses a scorer, rules, verifier, or human standard to choose the best one. It is not about having the model generate only once, but about spending more computation at request time in exchange for higher quality.

Best-of-N Sampling · Inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The pottery workshop fires several cups at once, then has a cup inspector pick one

The lantern riddle shop originally ran on a set of familiar practices. The shopkeeper had his assistant bring him riddle faces as soon as they thought of them, and hanging riddles at the old pace rarely went wrong; the neighbors were happy to guess for fun.

Then things grew bigger. For the Mid-Autumn Festival, the cleverest riddles had to be put up, and the urging came from morning to evening. The newcomers thought it was a matter of quantity — just send more people and add more hours. But the veterans saw that many details had already crossed the boundary that the old methods could bear.

The person in charge tried the first method: have the clerks put an idea on the wall as soon as they thought of one. In the first few days, there was indeed visible improvement, the queue got a little shorter, and outsiders felt that this craft had finally caught up with the new scene.

But trouble quickly changed shape. Some people skipped inspections to rush progress, some forced unsuitable work into old processes, and the most skilled people were dragged down by trivial matters. Errors were not always visible on the surface and often only emerged after delivery. The old ledger could still explain where things came from, but it could not explain how to change for the next step.

The person in charge added more rules, requiring everyone to sign more, report more, and add another layer of protection. On paper it looked thorough, but on site it was slower; the craftsmen were busy proving they were not wrong, while the real cracks continued to widen in place. Onlookers only saw results fluctuating between good and bad, and people on site found it increasingly difficult to judge whether to save speed, quality, or responsibility first. Everyone felt they had done a little more, yet the whole thing did not become more stable because of it.

A silent old craftsman laid several failures on the table. He blamed no one, only had everyone look at the same detail: the new restriction had never entered daily practice, and only suddenly appeared before delivery. He laid the several reworks out in the order they occurred, and only then did everyone discover that mistakes were always amplified at the same corner.

So the workshop adopted a new rule: write multiple versions on the same topic, then have veteran readers pick the clearest one. At first everyone was uncomfortable, as if their hands and feet were tied; but after several rounds, rough shortcuts were blocked, and truly reliable moves gradually remained.

When the same pressure came around again, delivery did not become fancier, but it was clearly more reliable. The shopkeeper finally understood: trying several drafts before choosing can improve the quality of what goes on the wall, but it also uses more paper and ink. If you wait until the end to fix things, the cost is often already buried in every earlier choice; this new path did not eliminate cost, but it made cost show up early, so it could be handled while changes were still possible.

Reveal

This story is about: best-of-N sampling

Best-of-N sampling can also be understood as "selecting the best among multiple candidates": the system generates N candidate answers at once, then uses a scorer, rules, verifier, or human standard to choose the best one. It is not about having the model generate only once, but about spending more computation at request time in exchange for higher quality.

The cost is that cost and latency usually rise with the number of candidates, and the selection criteria must also be reliable. If the scorer is biased, the system may simply become more consistent at picking answers that look good but are actually wrong.

Metaphor mapping

  • A few cups: multiple sampled candidate outputs
  • The cup inspector: reward model, verifier, or scoring rubric
  • Firewood cost: extra inference tokens, latency, and compute
  • Different orders use different quantities: adaptive N based on value or difficulty
  • The guest sees only one cup: only selected answer is returned
  • Beautiful but not durable: weak verifier can select superficially good but wrong outputs

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

rejection samplingreward modelinference-time compute
京ICP备2026014272号