← Back to concept explainers

Concept Fable

Inference

Conclusion first:Here, inference refers to model inference or model execution: applying a trained model to new input and generating output. It is not logical reasoning in the human sense, but the model performing computation on the current request.

Inference · Inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The trained stoker facing an unfamiliar order for the first time

There was a furnace worker at the iron shop who had studied heat, iron color, and hammering for three years. The master took him to see thousands of knives, hoes, and hinges, and he finally could recite how each material should be handled.

On the day he finished his training, a strange order arrived from outside the city: a slender hook knife for a salt-mist sea vessel. There was no identical drawing in the book, and his master did not do it for him, only told him to make it on the spot according to what he had learned.

The stoker first dug out every pattern he had ever learned, hoping to find an identical diagram. When he couldn't find one, he nearly demanded three more years of training, as if only retraining could qualify him to fire the furnace.

The master reminded him: training is over; what lies ahead is applying the patterns learned to new input. You will not rewrite past textbooks, only produce a result in the current furnace.

The stoker began observing salt spray, ship ropes, grips, and the direction of force to decide fire temperature, hammering order, and cooling time. Every step consumed charcoal and involved waiting; work too slowly and the customer grew anxious; work too roughly and the blade would break.

Some orders can be heated several pieces at once, while others must be handled individually. The master had him record the time from receiving the order to the first hammer strike, total charcoal consumption, and whether the final product passed inspection.

A few months later, the shop discovered that the real bottleneck was not whether it could forge iron, but how to produce steadily between cost, speed, and quality once new orders arrived.

The stoker finally understood: learning a craft is one thing; applying that craft to every new request and delivering a result is another. The latter is the on-site capability the shop promises to the outside world every day. Later, when the stoker took on apprentices, he ran into the same misunderstanding. One apprentice thought that as long as he memorized the old drawings, he could handle all orders; another thought that whenever he encountered an unfamiliar piece, he had to go back and relearn from scratch. The stoker had each of them forge a deformed hook knife and record the whole process from reading the drawing, selecting material, and heating to delivery. Only then did everyone see clearly that on-site output depends on learned patterns but is also constrained by the current charcoal fire, the waiting queue, and quality requirements. Later the shop also recorded the charcoal consumption and rework for different orders. The stoker no longer pursued only getting it right the first time; he also had to know how much each on-site job cost. Every new order is a real on-site use. The shop's promise rests on this step. Every delivery consumes real resources.

Reveal

This story is about: reasoning

Here, inference refers to model inference or model execution: applying a trained model to new input and generating output. It is not logical reasoning in the human sense, but the model performing computation on the current request.

In LLM systems, inference includes prompt processing, prefill, decoding, sampling, batching, serving latency, and cost control. Usually it does not update model weights; it only uses an already trained model to give the current answer.

Metaphor mapping

  • Three years of apprenticeship: model training
  • Unfamiliar order: a new input or prompt
  • Lighting the furnace on site: running inference
  • Cannot rewrite the textbook: ordinary inference does not update model weights
  • From order taking to first hammer: time to first token / prefill latency
  • Total charcoal consumption: compute cost and token cost

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

servinglatencydecodingbatch sizeprompt

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号