Fable
Two noodle shops: one cooks as customers arrive, the other waits until all are gathered before putting them in the pot
In the old city there were two noodle shops across the street from each other. Shop A's big stove could cook eight bowls at once, but Old Song's rule was to cook one bowl at a time as it was ordered—when a slot on the stove opened, a new bowl went on; early arrivals got served early and left satisfied. Shop B's big stove also held eight bowls, but Old Tian's rule was to wait until eight bowls had accumulated before firing the stove—early customers waited for later ones, and nothing cooked until the stove was full.
Porters in a hurry all went to Hall A. As soon as a bowl of noodles hit the pot, it was cooked in the time it took to turn around, and they carried it off—the eight slots on the stove always looked full, with old bowls taken away and new ones added. The porters called this "quick in, quick out—the stove waits for no one." At Hall B, a porter waited nearly two incense sticks' worth of time, and the stove was still three bowls short of the count. Old Tian was not anxious, saying that only when eight bowls were filled would the first batch taste right.
But another type of customer got a better deal at Hall B. When hosts held banquets, they ordered forty bowls of noodles at once. Hall B's large stove worked in five batches, eight bowls per batch, with high stove efficiency, and Old Tian could serve a whole table at once. Hall A cooked one bowl at a time; by the time it reached the nth bowl, the stove always had something going in and out, but forty bowls of noodles took nearly an hour intermittently—because the next person sat down only after the previous one finished eating, and several stove slots were occupied by slow-eating customers.
Old Song and Old Tian were called to debate at the chamber of commerce. Old Song said my stove is producing at all times; there is no situation where the stove sits empty waiting for customers. Old Tian said every time my stove fires up, it runs at full capacity; no burner is occupied by one bowl of noodles and slowing down the whole operation. The people at the chamber could tell that both sides were doing batch processing, but one prioritized online latency—the speed of getting food to the table—while the other prioritized offline throughput—how many bowls can be produced per unit of time.
What really decided the winner between the two sides was a mixed order. A caravan placed orders with both, requiring the porter's fast food to come out early and the banquet's cold dishes not to wait too long. Old Song's fast food was a full two times faster than Old Tian's, but the banquet dishes actually reached the table more slowly than Old Tian's — because the stove slots were occupied by fast-food customers, and the banquet bowls could only squeeze into the remaining slots and come out slowly. Old Tian's banquet dishes were all served at once when complete, but three porters, waiting for the order to fill up, got so hungry they went elsewhere.
Later the guild gave classification advice. When customers do not need to wait synchronously and only care about how many bowls one stove can produce in a whole day, use Old Tian's way: fill a pot and then cook, maximizing stove utilization. When customers need to eat as soon as they arrive and leave as soon as they finish, use Old Song's way: enter the slot on arrival and release on completion.
The two noodle shops later added a line to their menus: cooking when a guest arrives is called continuous batching, while cooking once enough have gathered is called batch inference. The stove is the same stove; how the fire is lit determines who waits and how long.