Concept Contrast Fable

Context recall and faithfulness

Conclusion first:Context recall asks "were the relevant documents found?" Faithfulness asks "does the generated answer match the documents?" High recall only means the materials were found comprehensively; it doesn't guarantee the answer wasn't fabricated. These two metrics are only analytically valuable when viewed separately—combined, they yield a conclusion that is flawless but useless.

Context recall asks "were the relevant documents found?" Faithfulness asks "does the generated answer match the documents?" High recall only means the materials were found comprehensively; it doesn't guarantee the answer wasn't fabricated. These two metrics are only analytically valuable when viewed separately—combined, they yield a conclusion that is flawless but useless.

Fable

The person who finds the material and the person who writes the report

There are two roles at the research institute: the archivist and the report writer. The archivist's job is, upon receiving a topic, to go to the archive and find all relevant literature, placing it on the report writer's desk. The report writer's job is to read this literature and write a synthesis report.

Old Su, the head of the materials team, had been watching one metric for half a year: the researcher's "find-everything rate." For every project, all known relevant literature had to appear on the reporter's desk. The researcher performed very well, keeping the find-everything rate above 96 percent the whole time. Old Su was very satisfied with that number.

Until a client complained about a report. The report said, "Heavy metal content in the soil of this area exceeds the standard by three times." The client took the report to the environmental protection bureau, which sent people to measure on site and found no problem at all. The client came back and demanded of the institute: Where is your basis?

Old Su returned to the archive to check that batch of original documents and found that the archivist at the time had indeed found all the relevant documents: five papers, two monitoring reports, and twelve sets of soil sampling data, neatly laid out on the table. The problem was that the reporter had only looked through three of them, left most of the rest unread, and filled in the data from memory.

Old Su reviewed the monitoring. The data clerk found everything, but the report writer was sloppy. A high retrieval rate does not mean high report quality. Old Su realized that he had only managed the quality at the input end, while the quality at the output end was a complete blind spot.

Old Su added a new metric: every report must include a citation cross-reference table, and every data conclusion in the report must note which document and which page it comes from. During spot checks, three conclusions are selected at random and verified against the original documents. This metric is called the "factual accuracy rate."

A few months later, when the data came out, Lao Su discovered a pattern: topics with high recall did not necessarily have high faithfulness; but no topic with low recall had high faithfulness. This showed that finding materials comprehensively was only a necessary condition, not a sufficient condition. The two metrics each watched different stages: one watched retrieval, one watched generation.

Old Su wrote in the year-end report: after getting the system's recall data, don't rush to celebrate. Ask one question—it found everything, but did it say it correctly? Between these two questions lies an entire generation process. Skip one question, and under those beautiful numbers there may be a pile of fabrication.

Context recall

A metric for evaluating whether the retrieval stage found all relevant documents. It measures 'how much of what should be found was found,' caring only about retrieval completeness, not generation quality.

Faithfulness

A metric for evaluating whether generated answers are faithful to the retrieved document content. It measures whether what is generated is based on real sources, detecting hallucinations and fabrication.

Story Mapping

The researcher looks for documents: context recall—whether the retrieval step brings out all relevant documents.
Reporter writes the report: faithfulness—whether the generation step outputs based on retrieved content.
High recall but fabricated reports: the two metrics are independent, and high recall cannot guarantee high faithfulness.
Citation cross-reference table: a common practice in faithfulness evaluation—verifying each item back to the source.
When recall is low, faithfulness is also not high: recall is a necessary condition for faithfulness, but not a sufficient one.

In the project

RAG system evaluation should look at at least two dimensions: the retrieval side looks at recall, and the generation side looks at faithfulness. When faithfulness is poor, do not directly change the retrieval strategy—first check whether retrieval missed key documents or the model improvised on existing documents. The former is a recall problem; the latter is a generation control problem, and the place and method of fixing them are completely different. If you blend the two metrics into one composite score, problem localization will always be half a beat slow.

Context recall Faithfulness