Fable
The iron horse that runs errands for people must first learn to stop at the line of responsibility
West of the city there was a team of iron horses that could deliver goods, take inventory, and write receipts for merchant caravans. At first they only followed fixed routes, with cart drivers holding ropes beside them, so mistakes were easy to trace back.
Later the caravan handed more work to the iron horse: choosing routes, chasing final payments, judging cargo damage, and replying to customer complaints. Speed increased, and no one was willing to go back to the old way of queuing and waiting for people.
Trouble followed. The iron horse cut through narrow alleys to make time and crushed small stalls; it judged credit by an old ledger, so new merchants were always turned away at the door; it copied notes containing family matters into public receipts.
At first, the shopkeeper only draped a cloth reading "Acting in Good Faith" over the iron horses and made each horse recite three nice phrases before setting out. Accidents did not decrease; the reports simply contained more fine words.
One day, an iron horse mistakenly delivered the medicine shop's urgent package to the tavern. Only then did everyone admit that the problem was not whether the horse meant well, but who set the goal, who checked the impact, who could stop it, and who was responsible for the consequences.
The caravan redid its system: high-impact matters first underwent risk assessment, sensitive information was masked first, decisions involving persons and rights had to be reviewed by a human, and deviations and incidents had to be recorded and held accountable.
The iron horses were still fast, but their speed was placed within boundaries. Every route had a reason, every privilege escalation had an alarm, and every affected person could find an entry point for appeal.
The city finally understood that the more machines can do the work, the less responsibility can be outsourced to machines. A truly mature arrangement is one where capability, boundaries, oversight and accountability all land together. Later, an even trickier job came along. On the surface it looked like just a bigger version of an old problem, but in fact it forced out all the gaps hidden in the process: who takes over first, who can make changes, how to return to the previous step when something goes wrong. Places that used to get by on familiarity and tacit understanding now all had to be spelled out. The person in charge did not urge everyone to be more careful; instead he walked the floor again. He had people record every handover, every failure and every remedy—which places were merely slow, which places would harm the results, which places would be hard to trace back once wrong. After the new rules were rolled out, the first few days were not smooth. Some complained about an extra check, some felt their old experience was being insulted. But when the next anomaly appeared, for the first time everyone could follow the traces to find where the problem had occurred, instead of standing around the result guessing at each other.