← Back to concept explainers

Concept Fable

Continuous batching

Conclusion first:Continuous batching dynamically schedules the tokens of multiple requests together while the model is running. When some sequences finish, new requests can enter the still-running batch without waiting for the entire batch of requests to finish.

Continuous Batching · Inference optimization

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The noodle shop's pot no longer waits for the whole table to finish before cooking new noodles

The dock noodle shop had a large pot. In the past, the staff would wait until ten bowls of noodles were ordered before cooking a batch, then scoop them all out together and wait for the next group of customers to fill another batch.

When there were few guests, this was neat. Later, some only wanted a small bowl, some wanted extra meat and eggs, some ate quickly, some were still waiting for soup. The big pot was often half empty, yet a long line formed at the door.

The shopkeeper first required that only ten bowls be enough to start the pot, and then gave number tags to late-arriving guests. It looked fair, but in practice it made even a bowl of quick noodles get dragged down by slow noodles.

One worker tried to force all guests' noodles out of the pot at the same time, and as a result some were undercooked and some were mushy. Batching is not about flattening different progress levels.

The new chef changed the scheduling: there was always noodles in the pot, and whichever bowl was ready was scooped out, with the freed-up spot immediately filled with the next guest's noodles. Each bowl still cooked at its own heat, but they shared the same pot.

He also grouped noodles with similar cooking times together, leaving small slots for quick orders and back positions for long-boiled orders. The pot no longer waited for the slowest person, and the line no longer waited for the whole batch to finish.

Soon, the noodle shop found it could serve more noodles per unit time, and the wait for customers' first bite became more stable. The hardest part was not cooking noodles, but letting orders of different lengths and progress flow through the same resource.

The shopkeeper understood that old batch processing suited orderly queues, but a real dock is always uneven. Only by continuously feeding new requests into the running batch could the expensive pot avoid idling. Later, the noodle shop looked at just two numbers: pot utilization and customer wait time. A scheduler that could protect both was truly suited to high-concurrency service. The new apprentice later understood that the noodles in the pot were not mixed into one lump; each bowl had its own moment of entry, heat level, and exit position. True craftsmanship is letting uneven orders share the hot pot without dragging each other down. The shopkeeper later trained the apprentice to watch the pot surface, not just the end of the queue. Where a slot had just opened, where a short order could fit, and where the whole pot would be slowed—all had to be judged in a few breaths. When the night market opened and customers arrived in even smaller batches, this arrangement proved its value.

Reveal

This story is about: continuous batching

Continuous batching dynamically schedules the tokens of multiple requests together while the model is running. When some sequences finish, new requests can enter the still-running batch without waiting for the entire batch of requests to finish.

This can improve GPU utilization and throughput, and is especially suited to LLM services with widely varying request lengths. But the scheduling policy must control the waiting time of individual requests, otherwise overall throughput rises while user experience may worsen.

Metaphor mapping

  • Big pot: GPU batch execution
  • Wait for ten bowls before cooking: static batching
  • Fast lanes held up by slow lanes: different sequence lengths cause head-of-line blocking
  • Empty slots immediately take new faces: continuous batching
  • Each bowl follows its own heat: each request has independent decode progress
  • GPU utilization: GPU utilization and tokens/sec
  • Customer waiting: latency and TTFT
  • Final insight: Continuous Batching keeps batches continuously flowing and uses scheduling to improve throughput, rather than letting requests be tied down by fixed batches

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

Dynamic batchingvLLMschedulerthroughputlatencyserving optimization

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号