Fable
The exam questions leaked into the workbook first
Spruce Academy selected accounting apprentices every year. In the past, the master set a batch of questions, and students calculated accounts, checked documents, and found errors on the spot; whoever was steady got into the accounting office.
As the academy became famous, more prep shops appeared. Students drilled questions every day, and the teacher was pleased, thinking everyone's accounting skills were getting stronger year after year.
On one exam, nearly the whole class scored full marks. The teacher at first thought his teaching had reached perfection, until one student wrote in the margin a note that should not have appeared beside the question.
It turned out this year's exam questions had long ago been copied into the practice book. The student had not mastered the changes in the accounts, but had seen the same ledger, the same typo, the same trap.
The teacher first shuffled the order of the questions. The score was still high because the answers and solutions had been memorized; changing the order was not the same as changing ability. When actually hired, the full-score student fumbled in the face of new caravans, new tax receipts, and new discounts.
The old accountant redesigned the exam. He took stock of which questions had already leaked and isolated similar ones; then he used a new ledger to generate unpublished samples, keeping a portion for use only in final acceptance. The academy finally understood: if the ruler is memorized in advance, what is measured is memory, not ability.
At first, scores dropped after the new exam, and students and prep shops complained. The teacher did not revert to the old questions, but instead clarified the boundaries between practice questions, mock questions, and official questions, and recorded who entered which booklet.
A few years later, the academy no longer blindly trusted sudden jumps in scores. If the source of the questions was unclean, the prettier the score, the more suspicious it was. A truly reliable assessment must let students face changes they have not memorized in advance. Later, when newcomers entered the trade, the old masters no longer began with abstract labels; instead they had them review that failure: where intuition had been relied on before, why the first patch was insufficient, and how the new arrangement kept risk in a place that could be checked. Only when they could explain these details clearly, and also point out when to keep using the method and when to upgrade it, were they considered to truly understand the approach. Later they also had newcomers rehearse with this method: first let the newcomer do it the old way and see how errors appeared; then redo it under the new rules and see which step was blocked in advance and which step left evidence. If newcomers could only recite names, they would still misuse it in similar scenarios; only those who could state the applicable boundaries, costs, and remedies after failure were allowed to take on work independently.