← Back to Concept Contrast

Concept Contrast Fable

Context precision and faithfulness: one measures retrieval relevance, the other measures whether the answer relies on evidence

Conclusion first:Context precision measures how much of the retrieved content is truly relevant, while faithfulness measures how much the model's generated answer is grounded in the retrieved content. Both metrics appear in a RAG evaluation report, but they point to different failure locations: one occurs in the retrieval stage, the other in the generation stage.

Context precision measures how much of the retrieved content is truly relevant, while faithfulness measures how much the model's generated answer is grounded in the retrieved content. Both metrics appear in a RAG evaluation report, but they point to different failure locations: one occurs in the retrieval stage, the other in the generation stage.

Fable

The apothecary has two scales: one weighs whether the herbs are right, the other weighs whether the prescription is accurate

The old pharmacy in the south of the city filled prescriptions from hundreds of medicinal materials every day. The assistant first brought possibly relevant medicines to the table, and the seated physician then wrote the prescription based on them. The shopkeeper felt this process was sufficient.

The shop later required that eight ingredients be presented to the master before every prescription. The proprietor thought that the more the master saw, the better the prescription. The assistants, looking for convenience, simply presented the eight most similar ingredients from the first row.

In the third month there was an incident. The patient actually had internal cold complicated by deficiency, yet most of the medicines on the table were for wind-cold, because the patient had once said "I caught a chill." The prescription the gentleman wrote was smooth and the medicinal properties did not clash, but it did not fit the real illness.

During the review, the shopkeeper set down two scales. The first scale weighed the assistants: of the eight ingredients brought, how many actually matched the illness? The more irrelevant medicines mixed in, the less precise the earlier search for medicine had been.

The second scale weighs the physician: can every ingredient in the prescription be supported on the table? If the right medicine is on the table but the physician swaps it for something else based on feeling, it shows that writing the prescription did not uphold the evidence.

In that incident, problems appeared on both sides. The clerk mixed in related materials, and the doctor replaced evidence-based aconite with dried ginger that felt safer to him. The pharmacy later investigated separately: the former was about finding the wrong medicine, so fix medicine-finding; the latter was about prescribing not according to the medicine, so fix prescribing. The two pipes cannot take the blame for each other.

The shopkeeper also discovered that good numbers in a report can deceive. If you only say that six of eight ingredients are relevant, people easily overlook that the two irrelevant ingredients may lead the physician in the wrong direction.

If one only says that the master had a source for every ingredient, it may still just be faithfully prescribing according to a bad batch of medicine. The apothecary later hung two scales side by side behind the counter. First check whether what was brought in was accurate, then check whether the prescription followed the evidence; only then would errors not conceal each other. Later, when newcomers were onboarding, the old masters no longer started with abstract terms, but had them review that failure: where intuition had been relied on, why the first fix was insufficient, and how the new arrangement kept risk in a place that could be checked. Only when they could explain these details clearly, and also point out when to follow the old way and when to escalate, were they considered to truly understand this approach. Later, they also had newcomers practice this approach: first let the newcomers do it the old way to see how errors emerged; then redo it under the new rules to see which step was blocked in advance and which step left evidence. If newcomers could only recite names, they would still misuse it in similar scenarios; only those who could state the applicable boundaries, costs, and remedies after failure were allowed to take on work independently.

Context precision

How much of the retrieved context snippets are actually relevant to the user's question. It locates the signal-to-noise ratio of the retrieval stage. A drop in precision indicates the retrieval strategy is stuffing in irrelevant content.

Faithfulness

The degree to which a model-generated answer can be traced back to the retrieved context. It pinpoints hallucinations or improvisation in the generation stage. When faithfulness drops, it means the model did not write the answer according to the evidence.

Story Mapping

  • First row of herbs: retrieved context chunks
  • The second row has been sifted by the gentleman: filtered or ranked search results
  • Third row of prescriptions: the answer finally generated by the model
  • The first scale (precision): measuring the relevance of retrieval results to the real question
  • The second scale (faithfulness): measuring whether the answer can be grounded in the retrieval results
  • Cold-symptom incident: precision was acceptable but faithfulness was very poor—the model did not generate according to the evidence
  • The two pipes carry different medicines: two metrics pinpoint different points of failure
  • New rule: precision drops, check retrieval; faithfulness drops, check generation

Down to RAG evaluation and quality monitoring

When building an evaluation system for a RAG system, do not look at precision and faithfulness on the same line. If users report that "the answer is inaccurate," first split the problem: go into the retrieval logs to check precision and see whether the returned chunks contain truly relevant information; then go into the generation logs to check faithfulness and see whether the model faithfully cited those chunks. Precision is optimized through retrieval strategy, chunk design, and embedding quality; faithfulness is optimized through prompt constraints, output validation, and citation mechanisms. When both metrics are low at the same time, retrieval usually collapsed first and generation could not hold it up.

京ICP备2026014272号