← Back to Concept Contrast

Concept Contrast Fable

Factual grounding and faithfulness—one governs telling the truth, the other governs not making things up

Conclusion first:Factual grounding measures the consistency between model output and external knowledge sources (documents, databases, search results). Faithfulness measures the degree of fidelity between model output and the given input or context—whether it has added things on its own beyond the given basis. Both look like quality metrics, but one looks outside and the other looks inside.

Factual grounding measures the consistency between model output and external knowledge sources (documents, databases, search results). Faithfulness measures the degree of fidelity between model output and the given input or context—whether it has added things on its own beyond the given basis. Both look like quality metrics, but one looks outside and the other looks inside.

Fable

There were two historians: one only checked whether it matched the original text, the other checked whether he had added his own words

The Hanlin Academy had two archive rooms: one collected external court bulletins and local memorials, and one kept the academy's own draft notes. The two historians sent out for field verification each had their own rules. Historian Zhao, after writing each passage, first opened the court bulletins and memorials to check—border grain prices, garrison organization, river water levels—whatever matched was called "stands up." Historian Qian did not look at the court bulletins; he only looked at the cited original text marked in his own draft notes—the draft said the border army had six battalions, but the original text only wrote four, so the extra two battalions were "added his own words."

First consolidated draft review: each of the two drafts has one passage flagged as anomalous. Historian Zhao's draft says "the river level rose to five chi two cun." Checking against the court gazette, the gazette indeed says five chi two cun—the fact is anchored. But the original annotated draft said "the water level was abnormally high," and Historian Zhao changed "abnormally high" to "five chi two cun"—the number matches the gazette, but not the meaning of the original. Historian Qian's draft left the number unchanged, but added a passage: "the water level was abnormally high, suspected upstream dam breach." The four words "suspected dam breach" appear in no source whatsoever.

In the second verification, the two historians swapped subjects. Historian Qian took Historian Zhao's draft to check against the court bulletin — all the numbers were accurate, full marks for grounding. Historian Zhao took Historian Qian's draft to check against the original text — faithfulness passed, with nothing added that was not in the original. The results of the two verification reports were completely different: the same draft was a fine piece by the grounding standard, but only half-rough by the faithfulness standard.

The scholar presiding over verification at the Hanlin Academy discovered the most dangerous case. Historian Zhao's draft had numbers that matched the court gazette, but the original memorial cited in the gazette had its numbers altered by the chief editor—the chain of anchoring was broken; the upstream was false data, and no matter how accurate the downstream numbers were, they were still wrong. Historian Qian's draft, though it added no extra content, had certain numbers that were actually vague descriptions in the original text calibrated by Historian Zhao using the court gazette—Historian Qian's fidelity standard could not see this source.

The scholar made a decision: from then on, verification would have two gates—the first gate, there must be no extra fabrication between the draft and the given source text; the second gate, the specific numbers and names cited in the draft must be traceable to external sources. Drafts that failed the first gate were returned directly; only after passing the first gate would they go through the second.

At the year-end overall review, the scholar wrote a summary: grounding checks whether "what you said is consistent with the external record," while faithfulness checks whether "you added things on your own to the materials given to you earlier." The two do not measure the same thing—a pair of matching drafts may be unfaithful to the source, and a draft faithful to the source may not match the facts. Quality evaluation needs two gates, walked in different orders.

Fact Anchoring

Measures the degree of consistency between model output and external factual sources (documents, knowledge bases, search results). It answers the question: "Can the output be traced back to a real external source?"

Faithfulness

Measures how faithfully a model's output adheres to a given input or context. It answers the question: "Is the output strictly based on the given material, without adding, inferring or fabricating on its own?"

Story Mapping

  • Historian Zhao verifies numbers against court bulletins: factual grounding—external knowledge sources validate output facts
  • Historian Qian checks additions against the original text: faithfulness — whether the output strictly matches the given input
  • The same draft is anchored at full score but only half-bold on faithfulness: the two dimensions evaluate different aspects of quality
  • The court gazette's original memorial data had been altered by the chief editor: when there is a problem upstream in the anchoring chain, the downstream anchoring results are also unreliable
  • The original vague description was calibrated by the court bulletin: faithfulness cannot see the source of calibration, but anchored interventions may change the meaning of the output
  • The scholar's two-gate verification rule: first check faithfulness (no fabrication), then check grounding (traceability); the order of the two gates matters

Down to quality evaluation for LLM applications

Evaluation of RAG or retrieval-augmented LLM applications needs to look at both factual grounding and faithfulness. First check faithfulness—whether the output answers within the scope of the given retrieval results, and whether it sneaks in the model's own knowledge. Then check grounding—whether the factual points in the output actually have clear sources in the retrieval results. The most insidious problem is when faithfulness passes but grounding fails—the system faithfully cites the retrieval results, but the retrieval results themselves are wrong; or when grounding passes but faithfulness fails—the system's knowledge covers the gaps in the retrieval results but mixes in content that cannot be traced. The two metrics should be reported separately and must not be merged into a single average score.

京ICP备2026014272号