← Back to concept explainers

Concept Fable

Online inference

Conclusion first:Online inference is calling the model in real time to generate results when a user request occurs, with typical constraints of low latency, high availability, concurrency capacity, streaming responses, and error handling. Online inference for enterprise Agents must also consider authentication, context assembly, tool calls, upgrade paths, SLAs, and observability.

Online Inference · AI engineering / operations

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The prescription at the counter cannot wait until tomorrow

The apothecary on the old street prepared common medicine packets in batches every morning, labeled them, and placed them behind the counter. For ordinary colds and bruises, the clerks could quickly hand them to customers.

Later the pharmacy took on an emergency window. Some people had night fevers, some had drug allergies, and some came with half an old prescription asking whether they could switch medicine immediately.

The shopkeeper initially still threw the question into the morning batch process. The clerk said to wait until tomorrow for a unified allocation, but the patient was standing at the counter waiting for an answer, and the risk grew with the waiting.

To look fast, the pharmacy also had clerks give off-the-cuff advice. After several near-misses, the shopkeeper realized that real-time response is not about speaking quickly, but about completing a reliable process within a short time.

He rebuilt the emergency counter: first confirm identity and contraindications, then check the pharmacy and old prescriptions, call a doctor when necessary, and finally hand the patient the recommendation and reasoning on the spot.

The emergency counter has its own constraints. If the line is too long, add staff; if key medicine is out of stock, downgrade handling; if the issue is high-risk, escalate; if the doctor cannot confirm, refuse to answer randomly.

The pharmacy also recorded every wait time, error, referral and patient feedback. Fast doesn't just mean fast on average; it also means not collapsing at peak times and having a fallback when things go wrong.

Later, ordinary medicine packets were still prepared in batches at dawn, while emergency cases went through a real-time counter process. The two methods each handled their own affairs and no longer dragged each other down. The proprietor understood that the core of online response is latency, availability, concurrency, and security boundaries. The person at the counter needs an answer now, but speed cannot be bought with loss of control. Later, an even trickier job arrived. On the surface it was just an old problem magnified, but in fact it forced out every gap hidden in the process: who takes over first, who can make changes, and how to return to the previous step when something goes wrong—places that used to be glossed over by acquaintances' tacit understanding now all had to be made clear. The person in charge did not urge everyone to be more careful again; instead, he walked through the site once more. He had people record every handover, every failure, and every remedy, noting which places were merely slow, which would harm the results, and which, once wrong, would be hard to trace back. After the new rules were rolled out, the first few days were not smooth. Some complained about an extra check; others felt their old experience was being insulted. But when the next anomaly appeared, for the first time everyone could follow the traces to find where the problem had occurred, instead of gathering around the results and guessing at one another.

Reveal

This story is about: online inference

Online inference is calling the model in real time to generate results when a user request occurs, with typical constraints of low latency, high availability, concurrency capacity, streaming responses, and error handling. Online inference for enterprise Agents must also consider authentication, context assembly, tool calls, upgrade paths, SLAs, and observability.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Emergency counter: online model service for user requests
  • Early morning batch processing: offline or batch inference
  • Off-the-cuff suggestions: fast responses lacking a reliable process
  • Identity, taboos, and old recipes: authentication, context, and business constraints
  • Call the doctor: tool calling, expert systems, or human escalation
  • Wait duration and peak performance: latency, availability, and concurrency metrics
  • Refuse to answer recklessly: safety boundaries and fallback
  • Final insight: online inference must simultaneously meet standards for real-time experience and engineering reliability

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

Real-time inferencemodel servinglatencyautoscalingSLAstreaming response

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号