Fable
Both noodle shops cook to order, but their rules for changing the pot and returning bowls are different
There were two noodle shops in town, both claiming they could cook customers' noodle orders together in one pot. Regulars thought they were about the same, since both saved fire, water and labor by cooking several bowls together. Only on market day, when customers lined up to the alley entrance, did the difference between the two shops show.
The stove in the south shop was always on. Before the first batch of noodles was fully out of the pot, the kitchen stuffed the newly arrived small bowl of noodles into the empty bamboo basket; when a customer ate quickly and the basket was freed, the next bowl was immediately added. There was always noodles cooking in the pot, and the stove did not wait for the whole pot to empty before starting the next round.
The north hall also looked at queue length to batch pots. When the manager saw ten customers arrive, he combined ten bowls into one pot; when the next batch arrived slowly, he combined fewer bowls to start a pot. Its batching changes with the arrival rhythm of customers, but each pot still rises and falls pot by pot.
An out-of-town boss came to inspect, heard both shops say "dynamic", and nearly signed the wrong contract. The master of the South shop held up his bamboo basket to show him: the key here is that the pot keeps filling in gaps while running, so old dough and new dough can coexist in alternation. The manager of the North shop said: my key is that before the pot boils I batch according to the current queue, and once it boils I run that batch through to the end.
That afternoon an urgent order came in. The south hall had just freed up a bamboo basket, so the urgent order was immediately placed into it, even though several old orders nearby had not yet gone out. The north hall had to wait for the current pot to finish before putting the urgent order into the next pot. The urgent order waited longer, but each pot's accounts were tidier.
The chamber of commerce later changed the register: continuous batching emphasizes continuously filling slots during runtime, letting slots be released gradually as generation ends; dynamic batching emphasizes assembling batches on the fly according to request arrivals, and the size can be adjusted before the batch opens. The two terms overlap, but "assembling a batch before it opens" and "filling slots during runtime" cannot be mixed into one thing.
From then on, anyone buying an inference engine would ask three questions: Can a new request be inserted into a running queue? Can a completed slot be released immediately? Is batch size decided before the batch starts, or does it keep changing during operation? Once these three questions are asked, it is clear who is selling which kind of stove.
Later, an out-of-town owner bought both stoves and tried running them, and only then did the account books really speak: the southern restaurant’s setup had fewer empty pots with scattered customer flow, while the northern restaurant’s setup was easier to account for with bulk takeout. The terminology was similar, but the businesses they suited were not the same.