← Back to concept explainers

Concept Fable

Tokens per second

Conclusion first:Tokens per second (TPS) measures how many tokens a model or service system generates per second, usually focusing mainly on the decoding stage. It is an important indicator of speed and throughput.

Tokens Per Second, TPS · Serving metric

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The scripture-copying house began counting how many characters fell per breath

The scripture-copying hall at the temple used to ask only how long it took to copy one scroll. The head of the hall looked at total time, the patrons looked at the delivery date, and no one cared how far the tip of the brush could travel on the paper in each breath.

Later, orders became more complex. Some scrolls required checking many footnotes at the beginning but were quick once actual writing began; others were simple at the start but slow on every sentence afterward. Looking only at total time, no one could tell where the slowness was. Customers began to complain: some scribes took a long time to start but wrote quickly once they began; others started immediately but wrote more and more slowly. The two kinds of waiting felt completely different.

The courtyard master first urged the copyists not to stop writing. The copyists said aggrievedly that sometimes they had not stopped; it was the earlier paper-finding, ink-grinding, and layout work that took the time.

So the accounting office began recording separately: when the first character appeared, how many characters on average fell per breath afterward, and which time periods had multiple people sharing one large desk. At first the accounting office recorded only the average characters per breath, which ended up including the preparation time at the beginning, making the speed appear to fluctuate. The copyists argued their case, and only then was the analysis split into segments.

They found that characters per breath could measure sustained writing speed, but could not by itself explain the whole experience. If the first character came too late, the guest would still be anxious; if the writing speed was high but the total scroll was too long, the wait would also be long.

The abbot also stopped comparing different scriptures directly. Some had large characters, some had many small annotations; when comparing speed, one must specify the paper, brushwork, number of concurrent scribes, and length of the scroll. Once they switched to new brushes, the character count rose, but so did errors. Only then did the abbot look at speed and error-correction cost side by side, so as not to chase only a pretty speed record.

With this metric, the scripture-copying institute can judge whether changing brushes, tables, or queuing methods actually made the generation phase faster, rather than just listening to whoever shouts that it's smoother. The director also uses this metric to make commitments. Urgent documents depend on first-token wait; long scrolls depend on sustained token output; many people at the same table depend on congestion. Reporting only one speed easily leads donors to misunderstand the actual delivery time.

The accountant finally wrote: characters per breath is not an honor plaque but an analysis table. It tells the scriptorium's sustained writing speed, but it must be read together with first-character latency, total scroll length, number of people at the same table and cost.

Reveal

This story is about: tokens per second

Tokens per second (TPS) measures how many tokens a model or service system generates per second, usually focusing mainly on the decoding stage. It is an important indicator of speed and throughput.

But TPS cannot be interpreted in isolation. Time to first token, total latency, prompt length, batch size, hardware, concurrency, and output length all affect the speed users actually perceive.

Metaphor mapping

  • How many characters fall per breath: tokens per second
  • When the first character appears: time to first token
  • Find paper, grind ink, set type: prefill and request setup
  • Continuous writing: decode phase generation
  • Many people sharing a large table: batching and concurrency
  • Paper, penmanship, number of concurrent users: benchmark conditions that affect TPS

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

throughputlatency

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号