← Back to concept explainers

Concept Fable

Throughput

Conclusion first:Throughput is the amount of work a model serving system can process over time, often measured as requests per second or tokens per second. It depends on batching, GPU utilization, KV cache memory, model size, sequence length, scheduling, and downstream tools. High throughput lowers unit cost and supports scale, but must be balanced against latency, fairness, and service-level targets.

Throughput · Inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The granary finally figured out how many bags of rice it can produce per hour

The disaster-relief granary opened its doors and handed out rice. The warehouse officer initially focused only on the fastest weigher: he could weigh twenty sacks in the time it took an incense stick to burn, so everyone praised the granary's efficiency. But the line outside only grew longer.

The new supervisor stood nearby watching for a day and found that even the fastest weigher often waited idle. The person fetching rice had not delivered the bag, the accountant had not recorded the village name, the sealing station lacked rope, and the loading entrance was blocked by a large cart. The fastest instant could not save the whole line. The warehouse official also used the fastest weigher as a model for others to imitate. As a result, everyone learned to weigh quickly, but the rice bags clogged even faster at the sealing station, and the elderly in line waited even longer.

The warehouse officer first added more weighers, but the weighed rice sacks piled up on the ground and the ledgers could not keep up. Then he added more bookkeepers, and the loading bay became even more congested. Making any single segment faster on its own can push the bottleneck to the next segment.

The overseer split the process into five stages: fetching rice, weighing, sealing bags, bookkeeping and loading. He looked at how many bags each stage could stably complete in a quarter hour, and also at who was waiting for whom. He merged identical village orders, interspersed short orders among long ones, split the loading dock into teams by route, and had the accounts office write village tags in advance. The first time, the overseer adjusted only the bottleneck and did not manage the pace at the entrance. Too many people were let into the rice-fetching area at once, and although weighing was fast, the area in front of the accounts desk fell into chaos again.

By the end of the day, noticeably more rice was being handed out, and the lines moved more smoothly. But the chief steward warned: chasing only the number of bags distributed could leave distant villages perpetually at the back. Throughput has to be viewed together with waiting and priority. He later added a few protective rules: emergency villages go first, long-distance convoys load early, and long-waiting lines get short orders inserted. Distributing more also means distributing without losing order.

After the disaster relief ended, the steward broke each day into segments for review. Mornings were stuck at bag pickup, midday at the accounting office, evening at loading. Bottlenecks move, and throughput must be watched continuously—you cannot only fix yesterday's most congested spot.

The warehouse official finally understood that throughput is not a performance of how fast any one person can be once, but how much the entire process can reliably deliver per unit of time. To improve it, you have to look at bottlenecks, idle waiting, and rhythm, not just applaud the fastest step.

Reveal

This story is about: throughput

Throughput is the amount of work a model serving system can process over time, often measured as requests per second or tokens per second. It depends on batching, GPU utilization, KV cache memory, model size, sequence length, scheduling, and downstream tools. High throughput lowers unit cost and supports scale, but must be balanced against latency, fairness, and service-level targets.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Granary: model serving system
  • Rice bags per hour: requests/sec or tokens/sec
  • The scale operator's fastest time: single-request speed, not system throughput
  • From fetching rice to loading the cart: pipeline stages in serving and Agent workflow
  • Merge village orders: batching and scheduling
  • Some villages wait too long: throughput-latency tradeoff and fairness
  • Final insight: Throughput determines system scale and unit cost, but it cannot be optimized in isolation from latency and SLA

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

latencybatchingGPU utilizationservingtokens per second

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号