← Back to Concept Contrast

Concept Contrast Fable

Fixed selection of a few versus selection by cumulative probability

Conclusion first:Top-k sampling randomly selects from the top k words with the highest probability, with a fixed candidate set size. Top-p sampling dynamically determines the candidate set size by a cumulative probability threshold—the more concentrated the probability distribution, the smaller the candidate set; the flatter it is, the larger the candidate set. Both control generation diversity, but one uses a fixed number and the other an adaptive threshold.

Top-k sampling randomly selects from the top k words with the highest probability, with a fixed candidate set size. Top-p sampling dynamically determines the candidate set size by a cumulative probability threshold—the more concentrated the probability distribution, the smaller the candidate set; the flatter it is, the larger the candidate set. Both control generation diversity, but one uses a fixed number and the other an adaptive threshold.

Fable

Two peasant women sorting beans: one grabbed a handful each time, the other looked at the size of the beans

The bean shop in town had two farm women responsible for sorting beans to boil into soy milk. Aunt Zhang's rule was simple: keep only the ten largest beans each time, regardless of what proportion those ten represent in the whole pile. Aunt Li didn't count a fixed number; she first looked at the overall size of today's bean pile, then kept a group of beans sufficient to represent the main flavor.

The first batch was top-grade northeastern soybeans, plump and even. Aunt Zhang kept ten beans, Aunt Li kept only four or five, and the two pots of soy milk tasted about the same. When the bean pile itself is concentrated, the difference between a fixed number and tasting for the main flavor is not obvious.

The second batch of goods was mixed beans, ranging in size from mung beans to broad beans. Aunt Zhang still kept the ten largest, leaving almost only Northeast soybeans in the pot. Aunt Li kept the beans that together represented most of the flavor; there were many more of them, and the taste was closer to the original pile of mixed beans.

The shopkeeper initially praised Aunt Zhang for her consistency: ten scoops every day, easiest to keep accounts. But customers soon complained that on mixed-bean days the soy milk tasted like a different recipe. A fixed number made the rule simple, but when the bean pile was scattered it discarded the middle flavor too early.

On the third day, a few large stones got mixed into the bean pile. Aunt Zhang's ten beans nearly included stones, because they too were ranked near the front. Aunt Li first checked whether the main flavor was already sufficient, then removed anything that clearly should not go into the pot. Both had to guard against debris, but they guarded differently.

The manager wanted to set a uniform rule too: from now on, always leave twenty beans. Aunt Li shook her head. When the beans were concentrated, twenty was too wide and would bring in extra off-flavors; when the beans were scattered, twenty might not be enough. The problem was not whether the number was large or small, but whether the rule flexed with that day's distribution.

Later the bean shop kept both methods. When stability and easy control were needed, they used fixed counts; when adjustment according to ingredient strength was needed, they looked at cumulative main flavor. The shopkeeper also understood that a larger selection range isn't always better—the key is not letting a fixed threshold shut out possibilities that should be kept, too early.

Later, when a banquet came up, the shopkeeper would first ask the guest whether they wanted consistency or to preserve flavor variation. If every bowl should taste similar, leave fewer edge beans; if the day's bean pile's layers should show through, let the retained-bean range expand with the main flavor. What the two farm women argued about was never who was better at picking beans, but which threshold suited that day's pot. The bean shop also stopped treating "how much to retain" as an isolated number, and instead looked at it together with the day's concentration of the bean pile. A threshold that doesn't change with circumstances will seem stubborn on certain days.

Top-k sampling

Randomly select from the top k words with the highest probability. k is a fixed integer, and the candidate set size is unaffected by the probability distribution. Simple and intuitive, but it may miss reasonable options when the distribution is flat, and may include low-probability noise when the distribution is sharp.

Top-p sampling (nucleus sampling)

The candidate set size is dynamically determined by a cumulative probability threshold p. When the distribution is concentrated, the candidate set is small; when the distribution is flat, the candidate set is large. It has strong adaptive capability, but under extreme distributions the candidate set size fluctuates greatly.

Story Mapping

  • Aunt Zhang always grabs ten beans: Top-k sampling, a fixed candidate set that doesn't change with the distribution
  • Aunt Li looks at the bean pile to decide how much to keep: Top-p sampling, dynamically determining candidate set size by cumulative probability
  • Two pots of good beans are about the same: when the distribution is concentrated, Top-k and Top-p perform similarly
  • Mixed bean piles differ noticeably: when the distribution is flat, Top-k may discard a large number of reasonable candidates
  • Pebbles mixed in: fixed k is sensitive to noise, while Top-p can adaptively adjust to filter out noise
  • Using both methods together: first Top-p to set the range, then Top-k to take a fixed number, is a common combination

Down to designing generation parameter combinations

Top-k and Top-p can be used independently or in combination. When combining, doing Top-p pruning first and then Top-k truncation is a common order—let the probability distribution determine the range first, then use a fixed count as a final constraint. Using Top-k alone suits scenarios requiring stable, predictable output, but the k value needs to be adjusted based on your specific distribution. Using Top-p alone suits scenarios with high input diversity, and a p value around 0.9 is a reasonable starting point for most tasks. When testing, you should use actual business data to look at the distribution of candidate set sizes, rather than just using default values.

京ICP备2026014272号