← Back to concept explainers

Concept Fable

Greedy decoding

Conclusion first:Greedy Decoding selects the highest-probability next token at every step. It is fast, deterministic, and low-cost, suitable for fixed-format, low-creativity, short-output tasks. But it tends to converge prematurely, producing monotonous text, and local optima can lead to a poor overall answer. Enterprise Agents should use it for clear, verifiable, low-risk outputs, not as the default answer for complex reasoning or creative tasks.

Greedy Decoding · Decoding

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

At every intersection, choose the brightest light

The mountain city publishing house hired a poem-copying poet to continue couplets for customers. With every character the new poet wrote, he picked from his mind the one that felt smoothest and most fitting at that moment. The first line was often beautiful, and bystanders praised his speed.

But when he could write long couplets, the problem appeared. In the first half, he kept choosing the smoothest-sounding characters, and by the second half there was no rhyme left to match; earlier, for momentum, he used "ten thousand miles," and the small courtyard scene later could not connect. Every step was smoothest, but the whole couplet was not necessarily best. The client praised his speed the first time, but rejected the draft the second time. Every character in the first half sounded smooth, but together they blocked off the paths the second half could take.

The innkeeper first asked him to write more slowly. The poet-copyist still chose the smoothest character at each position, only more carefully, and the result was just as likely to narrow the path early. The old gentleman showed him the discarded drafts: the fault was not that some character was obscure, but that at every earlier step he had greedily taken the nearest convenience and left no room for what came later. The night he wrote more slowly, he hesitated longer at each position, still looking only at the best option right in front of him. The old gentleman said, hesitation is not foresight; foresight must also place on the desk the paths that might connect later.

For some urgent orders, the shopkeeper still had him write this way. Door notices, short congratulatory messages, lost-item notices—the most fluent next word is usually enough, and it's fast. But for marriage certificates, epitaphs, and long poems, several writing approaches must be retained, then the whole piece reviewed. The shopkeeper also posted a division-of-labor sign: urgent notices use the fast method, long poems list candidates first. Different tasks switch to different writing methods, so the bookshop doesn't gamble the whole piece on saving time.

The poem-copying scribe later set rules for himself: for short lines, one character can fix one character; for long pieces, leave room first; if the preceding text already constrains too much, better to go back and revise. The fastest way of writing is not foolish, only narrow in applicability.

The poet-copyist still liked the fast method. The shopkeeper did not forbid it, only had him first judge the length and cost of the work. If a mistake could be rewritten immediately, go fast; if earlier text would lock in later text, leave more paths open.

The publishing house learned that gradually choosing the next step that looks most similar right now brings clean, stable, and cheap results; the cost is that it does not look back, nor does it think much about future complex structures. When using it, you need to know that you are trading for speed, not guaranteeing global optimality.

Reveal

This story is about: greedy decoding

Greedy Decoding selects the highest-probability next token at every step. It is fast, deterministic, and low-cost, suitable for fixed-format, low-creativity, short-output tasks. But it tends to converge prematurely, producing monotonous text, and local optima can lead to a poor overall answer. Enterprise Agents should use it for clear, verifiable, low-risk outputs, not as the default answer for complex reasoning or creative tasks.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • The brightest lamp: the token with the current highest probability
  • Choose the brightest at every step: the locally optimal strategy of greedy decoding
  • Short confirmation letters: suitable for deterministic tasks with greedy decoding
  • Urgent items arrive late: local optimization leads to poor overall output
  • Limit usage scenarios: choose decoding strategies by task risk

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

decodingargmaxbeam searchdeterministic generation
京ICP备2026014272号