← Back to Concept Contrast

Concept Contrast Fable

Adjust the entire distribution and keep only the part where probability is concentrated

Conclusion first:Temperature applies a global scaling to the generation probability distribution, changing overall determinism and randomness. Top-p sampling keeps only the candidate words whose cumulative probability reaches the threshold, cutting off the low-probability tail. The two parameters often appear together, but one reshapes the distribution while the other performs truncation—tuning them together can make model output behave in unexpected ways.

Temperature applies a global scaling to the generation probability distribution, changing overall determinism and randomness. Top-p sampling keeps only the candidate words whose cumulative probability reaches the threshold, cutting off the low-probability tail. The two parameters often appear together, but one reshapes the distribution while the other performs truncation—tuning them together can make model output behave in unexpected ways.

Fable

The tea shop has two managers, one adjusts the heat, one sifts the tea leaves

The tea shop in the east of the city had two managers overseeing tea roasting together. Old Zhou handled the heat—he had a brass knob for temperature control; turn it left and the fire grew smaller and the tea tasted lighter, turn it right and the fire grew larger and the tea tasted stronger but burned easily. Old Wu handled sifting—the roasted tea leaves were poured into a bamboo sieve, and he set a mesh size; broken leaves smaller than the mesh were sifted out directly, and only what remained on the sieve surface was tea fit for the shelf.

On the new apprentice's first day on duty, Old Zhou taught him to control the heat, and Old Wu taught him to set the sieve mesh. The apprentice learned both, then went to the stove and stir-fried a batch himself. He turned the heat to maximum and set the sieve to the smallest mesh—he wanted the tea flavor to be the strongest and only the most intact leaves. After it came out of the wok, seven or eight large leaves remained on the sieve, and the broken tea leaves that had been sifted out covered the floor. The yield of the whole batch was less than ten percent.

The clerk thought a tea shop was cheating him. He asked a shop that claimed a ninety percent tea yield and found that its heat was set in the middle and its sieve holes were relatively large. With moderate heat, fewer leaves broke; with larger sieve holes, some broken leaves still passed through. The price of high output was occasionally tasting fragments in the tea soup.

Old Zhou called his apprentice to the stove and drew a diagram: the heat knob controls the flavor range of the whole pot—low heat makes the whole pot weak, high heat makes the whole pot scorched; this is global. The sieve holes control the elimination line—raise it and the completeness requirement screens out more broken leaves, lower it and more leaves pass through; this is local trimming. Global parameter tuning and local trimming parameters can't both be turned to extremes at the same time; they squeeze against each other.

The shop later took custom orders from out of town. Someone wanted extremely light tea—Old Zhou turned the heat to the minimum, and the sieve size did not matter, because whether the leaves were broken or not, the taste was light. Someone wanted extremely strong tea with no broken bits—Old Zhou turned the heat up, Old Wu set the sieve to the finest mesh, the yield dropped to twenty percent, and the price per bowl rose fivefold. The two sets of parameters could be matched, but the cost of that match had to be calculated in advance.

The most common mistake the assistant made was tightening the sieve holes when the tea flavor was already very weak—the heat had already made the whole pot weak, and the sieve then filtered out the few remaining good leaves, so the tea served was both weak and broken. Old Zhou called this parameter stacking kill: two seemingly unrelated knobs turned wrong in the same direction, and the tea shop had nothing sellable left.

Later the shop hung an iron plaque by the stove: the flame controls the whole pot, the sieve holes control the retention range. First adjust the flame to set the base tone, then set the sieve holes to control completeness. Twist both knobs at once, and even good tea leaves can be roasted to ash.

Temperature

Controls how smooth the model's output probability distribution is. The lower the temperature, the sharper the distribution and the more deterministic the output; the higher the temperature, the smoother the distribution and the more random the output. It globally scales the logits before softmax.

Top-p sampling (nucleus sampling)

Starting from the highest-probability words and accumulating, keep only the smallest candidate set whose cumulative probability reaches threshold p, cutting off the low-probability tail. The smaller the threshold p, the smaller the candidate set and the more conservative the output. It acts at a different stage from temperature.

Story Mapping

  • Old Zhou's temperature knob: temperature parameter, globally scaling the probability distribution, affecting overall randomness
  • Old Wu's sieve holes: the Top-p parameter, trimming the low-probability tail and controlling the size of the candidate set
  • Maximum temperature plus minimum sieve opening: extreme temperature and Top-p at the same time will cause output collapse
  • The shop with a ninety percent tea yield: a reasonable parameter combination balances diversity and quality
  • Very weak tea with an indifferent strainer: at low temperatures, Top-p has less effect on the output
  • Parameter stacking: the interaction effect of temperature and Top-p worsens output in extreme directions

Falls under generation parameter tuning

When tuning parameters, there is a recommended order: first set the temperature—temperature sets the tone, then adjust Top-p for tail trimming. When temperature is below 0.3, Top-p has little room to matter because probabilities are already highly concentrated. For scenarios that need diversity, prioritize raising temperature; for scenarios that need to avoid repetitive low-quality output, prioritize tightening Top-p. In production, it is recommended to run grid tests on commonly used parameter combinations to avoid pushing temperature and Top-p to extremes at the same time.

京ICP备2026014272号