← Back to concept explainers

Concept Fable

Latency

Conclusion first:Latency is the time a model or Agent system takes from receiving a request to returning a response, or generating tokens. It may include queuing, routing, retrieval, tool calls, prefill, first token, decoding, and post-processing.

Latency · Inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

At the ER door, what is measured is not how smart the doctor is, but how long the patient waited

The mountain town's clinic had several renowned physicians whose analyses were very accurate. The proprietor often told outsiders how skilled the clinic was—until a long line began forming at the emergency entrance, and the patients finally voiced their real complaints.

They don't care how elegant the prescription ultimately is, at least not when the pain is most urgent. They first ask when someone will respond to the first sentence, when they can get the medicine, and when they can go home.

The shopkeeper first plastered the walls with the credentials of famous physicians, then added consultation forms. The clinic looked more professional, but patients waited longer and grew visibly more impatient. Some gave up halfway and turned to the small shop next door.

Some only counted how many people were seen in a day and claimed high efficiency. But some of those patients waited two hours to see a doctor, while others waited only a moment. The average hid the real suffering.

A manager broke the process apart: from entry to response, response to analysis, analysis to medication, medication to follow-up. Each segment recorded typical waits, slow-tail waits, and worst-case scenarios.

They discovered that slowness was not only in the doctor's consultation. Registration, finding medical records, waiting for the medicine furnace, and reviewing contraindications could each make the overall experience worse. No matter how accurate the medical skill, if the first response comes too late, half of the trust has already leaked away.

The clinic began setting goals by scenario: emergencies get a response first, chronic conditions can wait for complete examination; simple prescriptions should be fast, complex consultations should explain the wait. Different patients were no longer shoved into the same line.

Later, when the clinic evaluated the new process, it no longer asked only whether the analysis was accurate, but also how long it took from the patient's knock to the first response and to the complete result. The structure of speed is part of service quality. The manager also looked separately at patients in the slow tail. The few who waited especially long were often precisely the most dangerous or most anxious patients. If one looked only at the average, the clinic would hide these people inside beautiful numbers. What truly affects reputation is often the worst stretches of waiting. The clinic later treated the first response as a medicine. It cannot cure the disease, but it lets the patient know they have already entered care; if this step is slow, later accuracy is hard to redeem. Later, during the review, everyone wrote this lesson into the daily rules: first see where on the site the result was truly changed, then decide what to keep, what to repair and what to limit. The patients felt it most clearly.

Reveal

This story is about: latency

Latency is the time a model or Agent system takes from receiving a request to returning a response, or generating tokens. It may include queuing, routing, retrieval, tool calls, prefill, first token, decoding, and post-processing.

Enterprise systems should look beyond average latency to percentiles such as p95 and p99. A small number of very slow requests are often the real cause of poor user experience and SLA breaches.

Metaphor mapping

  • Clinic: LLM / Agent service
  • Patient waiting: end-to-end request latency
  • First sentence response: TTFT or first meaningful response
  • Registration, medical records, medicine furnace: queueing, retrieval, tool calls, prefill, decode
  • Averages hide the pain: mean latency hides p95/p99 tail latency
  • Emergency and chronic illness: latency SLOs for different workflows
  • Final insight: Latency measures how long users wait, and must be managed by link and percentile, not just by looking at the model itself

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

throughputtime to first tokentokens per secondserving

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号