Concept Contrast Fable

Hallucination detection and fact verification: one checks whether the model is making things up, the other checks whether the answer is correct

Conclusion first:Many people treat detecting model hallucination and verifying whether an answer is true as the same thing. Hallucination detection focuses on whether the model's output is internally consistent and whether there are traces of fabrication; fact verification focuses on whether the output is consistent with an external knowledge base or the real world. Only when both metrics look good is quality truly up to standard.

Many people treat detecting model hallucination and verifying whether an answer is true as the same thing. Hallucination detection focuses on whether the model's output is internally consistent and whether there are traces of fabrication; fact verification focuses on whether the output is consistent with an external knowledge base or the real world. Only when both metrics look good is quality truly up to standard.

Fable

The two lamps of the quality inspection workshop

A precision parts factory hired a reporting clerk to write up the master's dictated production updates each day. He wrote fluently, and the shopkeeper liked him—until one report said: "Lathe No. 3 has been fitted with new bearings." The maintenance master was startled when he saw it; Lathe No. 3 had not been repaired at all.

The quality inspector first did one thing: check which sentences in the report looked fabricated. He circled "new bearings," "night shift completed," and "sufficient inventory" sentence by sentence, demanding the clerk produce sources. Those without sources were marked as suspicious statements.

But another report stumped him. The clerk wrote, "Lathe No. 2 produced one hundred and twenty pieces today." This statement had a source—it was said orally by the night shift—but the warehouse record showed only one hundred and eight. It wasn't fabricated, yet it could still be wrong.

The old factory director separated the two checks. The first checks whether the report contains unsourced, self-contradictory, or fabricated-sounding content; the second takes each key fact and verifies it against work orders, warehouse receipts, maintenance records, and master signatures. The former catches nonsense, the latter verifies truth.

Later, before reports were delivered, the quality inspection manual gained two more columns. The left column asked: Where did this sentence come from, and were details added out of thin air? The right column asked: Even if there is a source, is it consistent with verifiable evidence? Some sentences failed the left column; some passed the left column but failed the right.

The shopkeeper thought circling suspicious statements was enough and asked the clerk to delete those sentences. The next day the report became conservative, yet it could still miss key information that was true but had no written source; some sentences looked plain but the actual numbers were wrong. Judging only whether something looks fabricated cannot prove whether it is true or false.

So the quality inspector split the work into two processes. The first was to find assertions in the report that might have appeared out of thin air and remind the shopkeeper where to be careful; the second was to check item by item against repair orders, warehouse tickets, and the master's signature to confirm whether lathe No. 3 had actually been repaired and what the inventory really was. One was responsible for spotting suspicion, and the other for verifying facts. Both were needed, and neither could impersonate the other.

The clerk's reports thus became shorter but more reliable. The factory also stopped mistaking "sounds like it isn't making things up" for "facts have been verified." Catching signs of fabrication is only the first sieve; putting facts back into evidence for verification is what determines whether they can be written into the official work report.

Hallucination detection

Determine whether model output contains fabrication, contradiction, or inconsistency; the object of inspection is the credibility of the output itself.

Fact-checking

Compare model output with external knowledge sources or real data to judge whether the content matches objective facts.

Story Mapping

The first lamp checks for defects in the parts themselves: hallucination detection, checking whether output is self-consistent and whether anything is fabricated
The second lamp tests the mold: fact verification, checking whether the output is consistent with external facts
Composite scores hide the problem of what does not fit the mold: merged metrics make the true quality distribution invisible
The night watchman hears nonsense: hallucination detection first spots suspicious output
The accountant checked the books item by item: fact verification requires checking external facts one by one

Applying it to an Agent project

Don't use a single 'accuracy' metric to cover both hallucination and factual issues. Hallucination detection is better tracked with self-consistency and fabrication rate; factual verification requires an independent knowledge base reconciliation process. Set thresholds separately for the two types of metrics and trigger alerts separately, so that when quality problems arise you can see at a glance which line broke.

Hallucination detection Fact-checking