← Back to concept explainers

Concept Fable

Hallucination detection

Conclusion first:Hallucination Detection is a mechanism for detecting whether model output is inconsistent with the given context, source documents, or known facts. Common approaches include claim extraction, evidence matching, NLI/LLM judge, citation coverage checks, and manual spot checks. It fits well in high-risk pipelines such as RAG, customer service, knowledge Q&A, and compliance statements, but requires evaluating false positives, false negatives, and evidence base quality.

Hallucination Detection · Observability

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The ticket inspector does not look at loudness, only at the ticket stub

There was an announcer at the dock who could explain ships, cargo, and weather with great authority. Passengers heard him speak clearly and assumed all his information was reliable.

Once he reported the sailing schedule wrong. His voice was still loud and the format was intact, but the departure time on the ticket stub was completely different. By the time passengers reached the pier, the boat had already left.

The dock first had the announcer slow down and add repetition. Errors decreased somewhat, but key facts still didn't match the ticket stubs.

The new ticket inspector proposed a different set of checks: extract each key claim made by the announcer and verify it one by one against ticket stubs, port authority records, the weather station, and contracts.

Inspection results were divided into several categories: evidence supports, evidence refutes, evidence missing, overstated. This way the team could finally see error types instead of just saying 'that machine is unreliable.'

The ticket inspector is also connected to the production pipeline. Before high-risk answers are published, an automated comparison is run first; if inconsistencies are found, a rewrite, additional evidence or human handoff is required.

They also acknowledge that detection will miss cases and produce false positives. Incomplete evidence bases, inaccurate claim extraction, and complex synonymous expressions can all affect judgments. Therefore, detection results must enter evaluation and manual spot checks.

In the end, the dock managed a clear voice and factual reliability separately. Sounding true no longer meant it could be released. Later, an even trickier job arrived. On the surface it looked like an old problem magnified, but in fact it forced out all the gaps hidden in the process: who takes over first, who can make changes, and how to return to the previous step after an error. Places that used to get by on acquaintances' tacit understanding now all had to be made clear. The person in charge did not urge everyone to be more careful again, but walked the site once more. He had people record every handover, every failure, and every remedy: which places were merely slow, which places could harm the result, and which places, once wrong, would be hard to trace back. After the new rules were rolled out, the first few days were not smooth. Some complained about an extra check, and some felt old experience was being offended. But when the next anomaly appeared, for the first time everyone could follow the traces to find where the problem occurred, instead of gathering around the result and guessing at each other.

Reveal

This story is about: hallucination detection

Hallucination Detection is a mechanism for detecting whether model output is inconsistent with the given context, source documents, or known facts. Common approaches include claim extraction, evidence matching, NLI/LLM judge, citation coverage checks, and manual spot checks. It fits well in high-risk pipelines such as RAG, customer service, knowledge Q&A, and compliance statements, but requires evaluating false positives, false negatives, and evidence base quality.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Announcer: generate model output
  • Ticket stubs and port records: verifiable sources of evidence
  • Check key claims one by one: claim-level hallucination detection
  • Support/refute/missing/excessive: error classification
  • Automatic comparison before release: detection gates in the production pipeline

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

Factual consistencyFaithfulness evaluationSelfCheckGPTRAG faithfulness
京ICP备2026014272号