Fable
Not all work at the city gate can be handed to the first runner
The old city's service hall had only one long table. Whoever brought a request to the table, the duty officer handled it immediately. Changing addresses, checking inventory, writing contracts, chasing payments — all crammed into the same queue.
When the business first started, this was the fastest way. A customer sent a message, the front-desk clerk immediately picked up a tool to handle it, and the result came back right away. No one felt a more complex arrangement was needed.
Later the tasks grew heavier. Some requests only needed to look up a single record, some needed ten minutes of reasoning, and some needed to wait for an external system to reply. The front desk window got blocked by long tasks, and short tasks had to queue along with them.
The supervisor's naive fix was to send more runners. In the short term the queue looked shorter, but a few days later worse problems appeared: the same task was grabbed by two people, failed tasks were forgotten by everyone, and the backlog at night crashed the system together in the morning.
Someone again proposed letting the front-desk clerk decide the order himself. But the front-desk clerk only saw the request in front of him; he did not know the citywide load, merchant priorities, task dependencies, or failure counts. Clever local judgment cannot save a chaotic global schedule.
The new steward turned the long table into a task cabinet. Each task card clearly stated its type, tenant, priority, deadline, retry count, and executable conditions. The front desk was only responsible for taking orders and filing them into the cabinet; workers drew them from different drawers according to the rules. The task cabinet also reserved capacity for different types of work. The people writing contracts did not go move inventory, and the people chasing payments did not grab technical fixes; when the backlog grew too large at night, the dispatch clock first protected the short jobs that customers were waiting on.
Next to the task cabinet there was also a scheduling clock. Light tasks took the fast lane, heavy tasks went to the back office, failed tasks were retried according to a backoff time, and tasks with unfinished dependencies were frozen first. After each worker received a task, they had to register the start, completion or reason for failure. What satisfied the supervisor most was that failures no longer disappeared. Which card had been retried three times, which card was waiting for a reply from the outer city, which card had to be handed to a master craftsman—all could be seen on the cabinet face. The service hall finally knew where it was busy.
From then on, customer requests were no longer dragged down by long-running tasks. The service hall could smooth peaks, add workers, pause certain types of tasks, and see clearly from the pipeline which queue was backing up. What the service hall learned was not queuing itself, but how to split work out of the conversational moment and turn it into schedulable, recoverable, observable execution units.