← Back to Concept Contrast

Concept Contrast Fable

Retrieval precision and retrieval recall

Conclusion first:Precision and recall are the pair of metrics in retrieval systems most likely to hold each other hostage. Precision asks how many of the things you found are correct; recall asks how many of the things you should have found you actually found. Improving one often comes at the cost of the other—the more you stuff in, the more complete the search results, but the easier it is to drag in noise. A single number can look beautiful alone while masking a collapse on the other side.

Precision and recall are the pair of metrics in retrieval systems most likely to hold each other hostage. Precision asks how many of the things you found are correct; recall asks how many of the things you should have found you actually found. Improving one often comes at the cost of the other—the more you stuff in, the more complete the search results, but the easier it is to drag in noise. A single number can look beautiful alone while masking a collapse on the other side.

Fable

Two messengers in the city's small newspaper: one picks and chooses, one searches everything

The town newspaper hired two couriers specifically to gather news leads from the market. Old Shi was a veteran who had worked the market for over forty years. Every item he brought back was reliable, but among the mere five or six items he always missed some important ones. Xiao Gu was new and could bring back forty items in a morning, covering almost every corner of the market, but twelve or thirteen of them were either duplicates or inaccurate.

At first, the editor-in-chief only paid attention to Old Shi, because Old Shi's drafts did not need review and went straight to the page. He always read Xiao Gu's drafts once and deleted them once, complaining that it wasted time. But in one issue reporting counterfeit goods at the market, Old Shi reported three problematic vendors, while Xiao Gu reported eleven—and the later investigation confirmed that all eleven indeed had problems, while Old Shi had missed eight. One of those eight later had a poisoning incident, and the newspaper was sharply criticized.

Only then did the editor-in-chief understand that Old Shi's problem was not that he searched incorrectly, but that he never went to those corners at all. His precision was close to 100 percent, but his recall was less than 30 percent. Xiao Gu's recall was over 90 percent, but his precision was only 60 percent. Replacing two couriers with one meant either being safe but missing big things, or being comprehensive but reviewing manuscripts every day.

The editorial department held a three-day meeting and reassigned the two couriers. Old Shi handled verification—Xiao Gu brought back forty leads, and Old Shi picked out the most reliable parts that did not need further review, while marking others that needed a second round of confirmation. Xiao Gu only handled searching, with no limit on whether leads were repetitive or rough, but every lead had to include a source rating.

The new layout process also changed. The front page only published news verified by Old Shi, with a small list of all clues found by Xiao Gu attached behind it, marked as pending confirmation, confirmed, or ruled out. Readers could judge the source level themselves, but the newspaper was only responsible for the verified parts.

The newspaper's approach was discussed by peers for a long time, but half a year later no one was arguing about whether it should be done—everyone was arguing about what precision to set and what recall to set. Each newspaper chose a different balance point based on its own manpower, budget and reader tolerance.

The editor-in-chief later told new editors just one thing: precision is the money you spend to protect your credibility, recall is the money you spend to avoid missing things. These two sums of money cannot be each other's budget.

Retrieval precision

The proportion of relevant items among the results returned by retrieval, measuring the system's ability to avoid outputting irrelevant content—the higher the precision, the less noise users see.

Retrieval recall

The proportion of all relevant entries successfully retrieved by the retrieval system, measuring the system's ability to cover all relevant content—the higher the recall, the fewer key pieces of information are missed.

Story Mapping

  • Old Stone (precision): every item picked was correct, but a large amount of key information was missed
  • Xiao Gu (recall): found almost all the clues, but mixed in a lot of noise
  • The Counterfeit Goods Incident: The Catastrophic Omission Caused by Pursuing Precision Only
  • Three-tier review: use high recall for initial screening and high precision for final screening in real systems
  • Different newspapers, different balance points: the trade-off between precision and recall depends on the scenario and budget

Applying it to Andy Ren Xu's project

Do not look at precision and recall in isolation. A good RAG retrieval pipeline usually first pulls back candidates with high recall, then uses reranking or filtering mechanisms to improve precision. When explaining metrics to clients, do not say precision is 95%; also say what the recall is. What clients really care about is whether your system will miss the key piece of information that makes them lose a lawsuit or a contract.

京ICP备2026014272号