Fable
Of ten arrows, exactly how many hit the bullseye
A new notice hangs at the gate of After-Sales City: knowledge base retrieval is now live. Each time a customer asks a question, the whole apparatus retrieves ten passages and hands them to the responder to write a reply. In the first week, everyone was excited. The apparatus almost always found some seemingly relevant passages. Looking at the table full of material, the supervisor said that since there are results every time, retrieval should be good. But customer complaints did not decrease. The responder said the material was there, but often only two or three passages were truly usable; the rest merely shared a product name, version number, or common phrasing. The craftsman first reduced the number of returned passages from ten to five. The desk was cleaner, but key evidence was sometimes deleted. Raising it to twenty brought more evidence, but the noise came back too.
So they began making a target verification table. For each question, they manually marked which passages truly answered the question, then calculated what proportion of the results returned by the whole setup were hits.
In one licensing issue, the entire system returned ten passages, but only four were truly relevant. In the past everyone just said 'we found a lot'; now they had to admit: only four of ten arrows hit the bullseye.
This number changed the discussion. The team no longer argued over feelings, but looked at how the hit rate changed under different coordinate methods, chunk sizes, and reranking strategies.
The supervisor also realized that high precision did not mean everything was fine. If only two passages were retrieved and both were relevant, the ratio was high, but key policies might be missed. Precision must be viewed together with recall.
From then on, acceptance of the entire retrieval apparatus was no longer about whether there were results on the page, but about how much of the context returned to the agency was evidence that truly should be seen.
The target-verification sheet also saved the responder a lot of arguments. In the past he would say the material couldn't be used, and the craftsman would say it had already been found; now they pasted each segment onto the target, and a hit was a hit, a graze was a graze.
After several rounds of adjustment, the supervisor began looking at two things at once: how much of the retrieved paper was actually usable, and whether truly important paper had been missed. The notice at the city gate was also changed; it no longer boasted "can find it," but promised "the evidence retrieved can withstand verification."