← Back to Concept Contrast

Concept Contrast Fable

Online inference and batch inference

Conclusion first:One requires millisecond-level response, with users waiting to see results; the other can queue and run for hours, with answers collected the next day. Choosing the wrong inference mode either explodes costs or leaves customers unable to wait.

One requires millisecond-level response, with users waiting to see results; the other can queue and run for hours, with answers collected the next day. Choosing the wrong inference mode either explodes costs or leaves customers unable to wait.

Fable

Instant kitchen and central kitchen

The riverside restaurant served walk-in guests while also making lunch for a hundred people at the academy. The cook, A-Gui, managed two seemingly similar operations: one where guests sat down and ordered, with dishes coming straight off the wok, and another where boxed meals were prepared in the morning according to a list. The new helper found it troublesome and merged the two ledgers into one, thinking that both were about success or failure anyway.

At first no one objected. The shopkeeper recorded both sides as "cooking," with walk-in customers also waiting for the big pot and lunch boxes also stir-fried one by one. Seeing fewer pages, the shopkeeper even praised them for saving effort. But once peak season arrived, the old accounts suddenly became unclear: was it that the previous stall hadn't finished, or that the next stall hadn't held the line? No one could answer.

They first added more seals and slogans and required everyone to write in more detail. As a result, the booklet grew thicker and disputes increased. When something went wrong, everyone could find a sentence to excuse themselves, but no one knew which end to save first.

The old steward called everyone into the courtyard and had them retrace the actual course of events. Whatever concerned the immediate action went into the first book; whatever concerned subsequent responsibility went into the second. The two books referenced each other, but were no longer mixed onto the same page.

At lunch, walk-in customers grew furious waiting, while the academy missed classes because batching was too slow. After separating them, many old conclusions changed: some work looked busy but did not actually block trouble; some arrangements seemed unremarkable in normal times yet preserved the whole in a disaster.

To prevent newcomers from mixing things up again, the old steward had them each handle a small case. Only when the two ledgers could each answer their own questions and also connect with each other at the boundary would the small case pass. If someone still wrote the two matters on one page, they had to return to the scene and look again at who acted first and who was ultimately responsible; when it was unclear, it was better to ask once more than let a muddled account stay overnight; because on the day something truly went wrong, one missing line of distinction would make an extra team work in vain.

Agui separated the front hall from the back yard: in the front hall, small pots stir-fried quickly; in the back yard, washing, cutting, steaming, and boiling were concentrated according to a checklist; both sides shared rice and flour, but scheduled work at different rhythms. Later, when newcomers entered the trade, the old steward had them remember only one sentence: if someone is waiting at the table, respond immediately; if a batch of names has long been fixed, gather them all and complete them in one go.

Online inference

Requests are processed immediately upon arrival, with latency in the millisecond to second range. Suitable for chat, search, real-time recommendations, and other scenarios requiring instant feedback, with a higher per-request cost.

Batch inference

Accumulate a large number of requests into a batch for unified processing, with latency at the minute-to-hour level, but greatly reduced unit cost. Suitable for offline labeling, report generation, data cleaning and other tasks that are not time-sensitive.

Story Mapping

  • Three minutes to serve in the front hall: online inference, low latency, high unit price.
  • Central kitchen cooking in large batches: batch inference, high throughput, low unit price.
  • One hundred group meal orders: large-batch tasks, insensitive to latency but sensitive to cost and consistency.
  • Sharing recipes and procurement channels: the same model can support both online and batch inference modes simultaneously.

In the project

When designing the architecture, split requests into two categories: real-time ones go through online inference, while reporting, analytics and batch-processing ones go through batch inference. The per-token cost of batch inference may be only 50% of online inference or even lower, but only if you can accept the latency. Also note: if the results of batch inference are to be used in real-time scenarios, an extra caching layer is needed.

京ICP备2026014272号