Fable
Many small switches connected into a judgment machine
The old scale in the city could only judge goods by one or two conditions: whether the weight exceeded a certain amount, whether the color passed. That was enough for simple goods, but not for complex ones.
Customers began asking to see texture, origin, shipping temperature, complaint history and contract tier all at once. Every rule on the old scale had to be written by hand, and the more rules were written, the more they looked like a tangled ball of string.
The engineer first added more knobs. Each knob corresponded to one condition. Once there were many knobs, no one knew how to combine them, and maintaining the rules became harder than the judgment itself.
A mechanic built a machine connected by many small switches. Each small switch receives some signals, passes them through weights and nonlinear transformations, and then passes the result to the next layer.
At first the machine knew nothing. During training, it adjusted connection strengths little by little based on errors. After seeing many samples, certain switches began responding to textures, certain ones to risk combinations, and later layers formed higher-level judgments.
When first used for preliminary contract risk screening, the machine combined clauses, amounts, regions, customer history and approval records, and output the probability that legal review was needed.
The legal lead reminded everyone that probability is not a verdict. Neural networks can learn patterns from complex signals, but high-risk outcomes must be traceable back to evidence, thresholds, and human review processes.
The team began recording training data, feature sources, threshold choices, and post-launch drift. The machine wasn't isolated intelligence—it was incorporated into a monitorable system. The city eventually called it the judgment machine: not because every switch understood the business, but because many simple units connected together could learn complex mappings. Later, a merchant asked the mechanic whether he could point to which small switch understood risk. The mechanic shook his head and could only show how the multi-layer connections changed together and which inputs most affected the results. So the city set acceptance rules for the judgment machine: it couldn't be treated as an oracle, nor as an ordinary scale. It excels at learning mappings from many signals, but the more it's used for important decisions, the more it needs monitoring, review, and exit options. Later the judgment machine also made mistakes, treating rare good goods as risky goods. The supervisor didn't smash the machine, but checked the samples, thresholds, and review process, and readjusted. Learned judgments must also be put back into the system. The more useful the machine, the more it must be watched.