← Back to concept explainers

Concept Fable

Time to First Token

Conclusion first:Time to First Token (TTFT) measures the time from when a request is sent to when the first genuinely generated token is received in a streaming LLM response. It determines how quickly the system appears to start responding.

Time to First Token, TTFT · LLM runtime / user experience

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The restaurant first serves a bowl of hot soup, so guests know the kitchen is working

The teahouse storytelling stage originally ran on a set of familiar methods. The storyteller would first strike the wooden block each night and then slowly unfold the story, and when opening on the old rhythm, things rarely went wrong; the tea guests were willing to wait for his buildup.

Later the matter grew bigger. The full house of guests began urging the first opening line, with the urging running from early morning to evening. The newcomer thought this was a quantity problem—just send more people and add more hours—but the veteran saw that many details had already crossed the boundary that the old methods could bear.

The person in charge tried the first approach: transcribe the entire book in the back office before speaking. In the first few days there was indeed visible improvement, the line got a little shorter, and outsiders felt this craft had finally caught up with the new scene.

But trouble quickly changed shape. Some people skipped inspections to rush progress, some forced unsuitable work into old processes, and the most skilled people were dragged down by trivial matters. Errors were not always visible on the surface and often only emerged after delivery. The old ledger could still explain where things came from, but it could not explain how to change for the next step.

The person in charge added more rules, requiring everyone to sign more, report more, and add another layer of protection. On paper it looked thorough, but on site it was slower; the craftsmen were busy proving they were not wrong, while the real cracks continued to widen in place. Onlookers only saw results fluctuating between good and bad, and people on site found it increasingly difficult to judge whether to save speed, quality, or responsibility first. Everyone felt they had done a little more, yet the whole thing did not become more stable because of it.

A silent old craftsman laid several failures on the table. He blamed no one, only had everyone look at the same detail: the new restriction had never entered daily practice, and only suddenly appeared before delivery. He laid the several reworks out in the order they occurred, and only then did everyone discover that mistakes were always amplified at the same corner.

So the workshop adopted a new rule: prepare the opening words, the character roster, and the first act first, then retrieve later text as you go. At first everyone was uncomfortable, as if their hands and feet were tied; but after a few rounds, rough shortcuts were blocked, and truly reliable moves slowly remained.

When the same pressure came again, delivery did not become fancier, but it was clearly more reliable. The storyteller finally understood: listeners feel speed from the moment the first note falls. If you wait until the end to fix things, the cost is often already buried in every earlier choice; this new path did not eliminate cost, but it made cost appear earlier, so it could be handled while changes were still possible.

Reveal

This story is about: time to first token

Time to First Token (TTFT) measures the time from when a request is sent to when the first genuinely generated token is received in a streaming LLM response. It determines how quickly the system appears to start responding.

TTFT is affected by queuing, routing, prompt processing, prefill, retrieval, tool calls, prefix caching, and prompt caching. It is different from total completion latency: a fast first token does not mean the entire answer will necessarily be fast.

Metaphor mapping

  • The first sip of hot soup: first generated token or first streamed output
  • From order to hot soup: TTFT
  • Reading the order and preparing materials: routing, retrieval, prompt processing, prefill
  • Pre-preparing the base stock for a fixed menu: prompt/prefix caching
  • Random side dish: fake progress that does not reflect real model output
  • Full banquet: total completion latency
  • Final insight: TTFT governs the wait for the first real response, and is critical to trust and usability in Agent products

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

LatencystreamingprefilldecodeKV cachep95 latency

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号