Fable
After Changing the Ink and Brush, the Accounting Office Recalculated the Old Accounts First
The Qingwa trading firm had a clerk who could write letters, check accounts, and draft contracts. In the early years, the manager only looked at whether the letters he wrote were fluent and whether customers nodded. When business was light, this was enough to get by. After the firm expanded, the clerk had to handle collections, quotations, contract exceptions, customer complaints, and internal approvals. It looked like writing, but in reality every matter involved money, responsibility, and risk.
The proprietor's naive fix was to casually test three letters after each change of pen and ink. If all three went smoothly, he assumed the new pen and ink were better. Then one quotation letter omitted a discount condition, and the firm lost a large sum.
A later review found that the clerk was more polished on ordinary greetings, but had regressed on complex clauses, factual citations, and boundary refusals. The three sample letters did not cover the truly dangerous work.
The old accountant proposed not to keep testing by feel. First list the most common, most valuable, and most error-prone tasks of the firm, then prepare examples, standard answers, scoring rules, and failure levels for each type of task.
From then on, changing ink, changing paper, changing documentation habits, or changing ledger sources all had to run the same test suite. Results looked not only at the final text, but also at facts, citations, format, risk, cost, and whether human takeover was needed.
Some tests are auto-graded, some need a veteran's eye, and some are deliberate traps. When scores drop, you can't wave it through with 'it reads more naturally overall'; when scores rise, you also need to check whether safety and cost were sacrificed.
The firm gradually formed a closed loop: online errors enter the sample library, the sample library enters the release gate, and the gate result determines whether to launch, roll back or continue adjusting.
What stung the shopkeeper most was that the letter about losing money never appeared in casual testing at all. Yet it was exactly the scenario the firm encountered most often and lost money on most easily. The shopkeeper finally understood that every time the writing-and-answering clerk changed, it had to pass the gate again. Quality had to become a system, not rely on a one-time acceptance check. Later, the shopkeeper also learned to look at the cost behind the scores. Once, a new clerk was faster on most questions but vague when refusing overreach requests; another time, new ink and brushes made letters prettier but reduced cited sources. The old accountant insisted on putting these into the same acceptance ledger. What the firm needed was not one beautiful trial draft, but the ability to explain, with every change, which capabilities improved and which risks grew. Later, these mistaken cases were no longer hidden as embarrassments but entered the next round of test questions. The more the firm feared a type of error, the more it had to put that error at the gate and test it repeatedly.