← Back to Concept Contrast

Concept Contrast Fable

Time to first token and tokens per second

Conclusion first:One measures how long after the user presses Enter before the first character appears; the other measures how many characters per minute flow once they start coming. Mixing these two numbers together to optimize may spend all the money on the wrong thing.

One measures how long after the user presses Enter before the first character appears; the other measures how many characters per minute flow once they start coming. Mixing these two numbers together to optimize may spend all the money on the wrong thing.

Fable

Open the door fast and move the goods fast

By the canal there was a warehouse run by a manager surnamed Zhou, specializing in transshipping goods north and south for the city's shops. The warehouse process had two steps: when the picker arrived at the door, he first handed over the slip, verified the goods number, and opened the warehouse door; then the workers went in and moved the goods from the shelves onto the carts. Manager Zhou had done this for over a decade, and the process was as familiar as the palm of his hand.

Several new warehouses opened on the opposite bank, with similar prices but promises of faster fulfillment. Manager Zhou's business began to decline. He gathered the clerks, raised each one's wages, and demanded that the speed of moving goods double. The clerks trained hard for two months; a cartload of goods that used to take three trips now took only two, each one drenched in sweat but moving with wind at their feet.

But the shops' complaints did not decrease at all. A cloth merchant found Manager Zhou: I sent my clerk to your warehouse to pick up goods, and when he arrived at the door he waited nearly half an hour before anyone came to open it. After opening, there was still registration, document checking, and seal stamping. Manager Zhou went to the warehouse door himself and squatted there for three days, seeing with his own eyes a long line of people waiting to pick up goods, while inside the warehouse the clerks, having finished moving goods, sat on crates chatting—the door was not open, so the goods could not get out.

Shopkeeper Zhou pulled out the ledger and found that he had put all his effort into moving goods faster, but the bottleneck was not in moving goods at all. From the moment the pickup slip was handed in to the moment the warehouse door opened, the average wait was one stick of incense. Within that stick of incense there were two links that needed improvement: the registration master was getting old and had to bring every slip close to his eyes to read slowly; and the registration desk had only one table, so when two groups of customers arrived at the same time they formed a long line, and the people behind could only stand there doing nothing.

Manager Zhou did two things: he added a table to the registration desk and assigned a young clerk specifically to sort the slips and check the item numbers; and he installed a pull cord connected to a bell at the entrance, so when the picker pulled the cord, those inside knew someone had arrived and could prepare the delivery slip in advance. From then on, the time from the picker handing in the slip to seeing the goods dropped from the time it takes to burn a stick of incense to less than a cup of tea.

At the year-end inventory, the warehouse business had returned to its original level. Manager Zhou hung a wooden sign at the entrance with eight words: "When the gate is fast, goods move fast; when the gate is slow, moving goods is wasted effort." Beside the ledger he posted a new rule: from then on, in the monthly efficiency report, the first line would record gate-opening time—from submitting the form to the warehouse door opening—and the second line would record goods-moving speed—from the warehouse door to goods being loaded. The two numbers were to be recorded and managed separately.

A peer came to learn from him and asked how to improve delivery. Manager Zhou said: delivery has two parts—door-opening time and cargo-moving time. Door opening is slow but cargo moving is extremely fast, yet customers only feel the frustration of waiting during the door-opening part. If you mix the two parts into one total number, you will desperately make the staff run faster, but the door that should open still opens slowly.

Later, at an industry gathering, he made an analogy: opening time is the time a tea customer sits down and waits for the teacup to be brought over, and moving speed is the speed at which the tea leaves steep. No matter how fast the tea steeps, if you do not bring the cup over for ten minutes, the customer has already gotten up and gone elsewhere. The two numbers must be measured separately: one governs the door, the other governs the goods.

Time to First Token (TTFT)

The latency between when a user request is sent and when the model returns the first token. This value includes request queuing, context preprocessing, prompt encoding, and KV cache initialization. It determines the user's first impression of the system's responsiveness.

Tokens per second (TPS)

The number of tokens a model can generate per second during the generation phase. It measures the pure computational throughput of the decoding process, and usually only takes effect after the first token. High TPS means long responses are generated smoothly, but it does not solve the problem of "how long until it starts."

Story Mapping

  • The clerk doubled the speed of moving goods, but customers still complained: moving fast does not equal being fast overall; the bottleneck is not here.
  • The opening time was a full incense stick long: slow registration clerks, a single table, queuing, the problem was all at the entrance rather than inside the warehouse.
  • Add tables, assign young helpers, install pull-cord bells: optimizing opening time does not require touching the handling step; the two processes are governed separately.
  • The efficiency report has two lines — door-opening time and cargo-moving speed are recorded separately: the two numbers follow completely different management logic.
  • Fast tea brewing and when the teacup arrives: a metaphor from everyday life that makes segmented latency immediately understandable.

In the project

On the monitoring dashboard, TTFT and TPS are tracked and alerted separately. When TTFT exceeds the threshold, check preprocessing and caching; when TPS exceeds the threshold, check GPU compute planning and batching strategy. Interactive products prioritize TTFT, while batch-processing products prioritize TPS. The optimization paths for these two scenarios are different, and an optimization plan that mixes them together usually ends up pleasing neither side.

京ICP备2026014272号