Fable
That all-purpose steam brain, even without opening the door itself, affects all door locks
The workshop city built an all-purpose steam brain. It could write letters, draw pictures, translate, keep accounts, and teach apprentices; connected to different tools, it could do different work. At first, the city lord regulated only the machines that directly controlled gates, money, and medicine. The steam brain was merely a foundation, and its maker said he did not know what downstream users would do with it. But a few years later, half the city's machines were connected to it. Door locks asked it whether to allow passage, the accountant asked it whether a transaction was suspicious, and the school asked it whether an essay was plagiarized.
Once the steam brain learned a wrong prescription, a dozen downstream small tools repeated it at the same time. The accident did not happen at the base factory's door, but it spread from the base capability.
The city began requiring makers of the all-purpose steam brain to take on base responsibility: explain the practice and capability boundaries, assess systemic risk for the whole installation, provide safety instructions, and support downstream compliance.
If the steam brain is particularly powerful and broadly applicable, even stricter testing is required, major incidents must be reported, and it must be prevented from being misused at scale.
Downstream workshops cannot push all responsibility onto the foundation either. They must apply their own risk controls and human oversight for their specific use cases.
In the end, everyone distinguished two layers of responsibility: the general foundation must explain what broad risks it can introduce, and the specific application must prove that it is controllable in a particular scenario.
At first the base factory felt wronged: they had not personally managed the gates, money or medicine, so why should they manage the consequences? The old judge had them look at the accident diagram. The mistakes of more than a dozen small workshops all grew from the same misunderstanding of the base, and no one could pretend to be just passing by.
In the end the city settled on layered accounts. Base-layer factories had to explain broad capabilities and shared risks, while integrators had to explain their specific uses and protections. Once responsibilities were separated, there was less arguing, and the links that truly needed fixing were easier to find. This layered responsibility kept base-layer factories from hiding behind unknown uses, and kept downstream workshops from pushing their own carelessness onto others. The more common the all-purpose base became, the more indispensable this layered account was. Without it, risk would spread through every access point.