← Back to concept explainers

Concept Fable

Dynamic batching

Conclusion first:Dynamic Batching, also called In-flight Batching, adds new requests into an actively running inference batch as other requests finish. It is closely related to continuous batching and is used by LLM serving systems to improve GPU utilization under variable request lengths. The scheduler must balance throughput gains against fairness, priority, memory pressure, and tail latency.

Dynamic Batching / In-flight Batching · Inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The ferry learned to take on the next batch of cargo while sailing

The Eguchi dock used large ships to carry cargo. The old rule was that once a ship was full, the hold was sealed, and the next ship would not depart until all the cargo had reached the destination. If the ship carried one heavy box for a long route, several light boxes for short routes had to wait alongside it.

As soon as peak season arrived, long lines formed at the dock. Light-cargo merchants complained that a box of urgent letters had to wait for an entire boat of tea bricks; heavy-cargo merchants were also dissatisfied because the cabin was often half empty while waiting for all same-route goods to gather.

The captain first divided the fleet more finely: small boats ran short routes, large boats ran long routes. But tides and cargo volumes changed daily; sometimes small boats ran empty, and large boats were still dragged down by slow cargo.

The old dispatcher changed the loading method. When the ship reached a small port midway, cargo that had arrived was immediately unloaded; if the freed slot was the right size, new cargo heading in the same direction was taken on. Every exchange had to be recorded on the cabin slot board: who got off at which slot, who filled which slot, and who would handle handover at the next stop.

At first the sailors loaded cargo randomly and nearly crushed fragile porcelain under ironware. So the dispatcher added rules: cargo type, destination, weight and priority had to match, and cabin changes could only be registered by the cabin-tag officer. Flow can improve turnover, but losing records turns it into chaos.

Mistakes happened in the first week too. A sailor, focused only on taking new cargo, nearly stuffed a box of urgent letters into a cabin slot on a detour route. So the old dispatcher set a rule: before swapping cargo, you must check the destination, remaining distance and cabin slot size; take it only if you can, and never take random jobs just to fill the hold.

The new loading method made the dock busier and also more in need of rules. Sailors had to watch the grid, the direction, and the remaining distance while the ship was moving, not just wait for the whole voyage to end. Light cargo was no longer tied down by slow cargo, and the hold no longer sat empty because of rigid batches. Later, at every mid-route small port, the captain was like a player in a game of chess: which goods had arrived, which slots were available, which new goods could be picked up along the way. The ship did not stop into dead batches; the voyage itself became part of scheduling.

After several trips, the boat no longer had to wait for the whole batch to share the same fate. Short-route goods disembarked on time, long-route goods continued onward, and empty cabins decreased. The captain understood that a batch does not have to be a doomed formation that goes all the way to the end; unloading, replacing, and rearranging with discipline while in motion is what allows throughput and order to exist at the same time.

Reveal

This story is about: dynamic batching

Dynamic Batching, also called In-flight Batching, adds new requests into an actively running inference batch as other requests finish. It is closely related to continuous batching and is used by LLM serving systems to improve GPU utilization under variable request lengths. The scheduler must balance throughput gains against fairness, priority, memory pressure, and tail latency.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Ferry: model serving resources
  • Whole-boat batches: traditional fixed batch
  • Unloading midway and empty cabins: releasing the batch slot after the sequence completes
  • Taking on new cargo while sailing: in-flight batching
  • Cargo labels: request length, priority, tenant, memory footprint
  • Hull balance: memory and scheduling constraints
  • Final insight: the point of Dynamic Batching is to fill gaps in running batches so the GPU waits less, but it must not sacrifice latency and fairness

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

continuous batchingvLLMTGIserving throughput

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号