← Back to concept explainers

Concept Fable

Early departure

Conclusion first:Early exit, also called early-exit inference, lets a model stop at an intermediate layer when confidence is high enough, instead of running the entire model every time. Simple samples use less computation, while difficult samples still take the full path.

Early Exiting · Inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The checkpoint no longer required every letter to pass through the entire gate tower

The north gate tower has twelve checkpoints. Every letter entering the city had to pass from the first gate to the twelfth, and finally be stamped by the chief guard. In early years there were few letters, so this seemed prudent. Later letters split into two types: one merely asked what time the market opens, the other concerned border defense and grain supplies. But both types had to go through the entire gate tower, and even the simplest question had to wait until the afternoon. The adjutant first added more runners to carry letters from one gate to the next. It was slightly faster, but each gate still had to check once, and short questions and short answers did not take fewer steps.

Someone else suggested putting a green tag on simple letters so the first few checkpoints could glance and release them. As a result, some people, in the interest of speed, tagged complex letters green too, and several times nearly missed important matters.

The old guard changed the rules: each checkpoint not only checked the content but also wrote down how certain it was. If the earlier checkpoints were already highly consistent, the letter could leave the city early; if opinions were scattered, it continued to deeper checkpoints.

This isn't laziness. Early release only happens when the letter is simple enough and the judgment stable enough. Grain, litigation, and border-defense letters still must pass through the later gates.

After a while, the average wait at the gate tower dropped a lot, and the hardest letters received more attention. The chief guard could also see from the records which letters sent out early later needed review.

The adjutant finally understood: saving time cannot rely on cutting several checkpoints across the board. The right approach is to let computational depth vary with question difficulty, reserving the full cost for the requests that truly need it.

The old guard also set up a review basket. A portion of the green letters that left the city early would be sent to the back gate for re-checking; if the front gate was found to be overconfident, the threshold would be raised. This way, leaving early wasn't a permanent privilege, but a judgment that was continuously calibrated.

The gate tower later reserved the time saved for difficult messages. Simple inquiries no longer had to squeeze through all twelve checkpoints, and complex military reports were no longer slowed by trivial matters. The adjutant understood: a smart gate is not one that always opens early, but one that knows when it can open early and when it must investigate deeply. Later, the gate tower came to see early exit as a right, and also as a promise that must be continually re-examined.

Reveal

This story is about: early exit

Early exit, also called early-exit inference, lets a model stop at an intermediate layer when confidence is high enough, instead of running the entire model every time. Simple samples use less computation, while difficult samples still take the full path.

It can reduce inference cost and latency, but only if confidence is calibrated and quality is continuously monitored. Otherwise the model may exit too early and answer seemingly simple but actually difficult questions incorrectly.

Metaphor mapping

  • Twelve gates: transformer layers or inference stages
  • Simple marketplace letters: easy queries
  • Provisions and border defense letter: hard or high-risk queries
  • Write down the level of certainty at each gate: confidence / exit criterion
  • Exit the city early: stop computation at an intermediate layer
  • Review records: monitoring accuracy and regression after early exit

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

dynamic computationinference optimizationlatency
京ICP备2026014272号