← Back to Concept Contrast

Concept Contrast Fable

Detecting hallucinations and the risk of evaluation hallucinations

Conclusion first:Hallucination detection catches false content that has already been produced, either after the fact or in real time; hallucination risk assessment judges in advance how likely a given scenario, model, or data combination is to produce hallucinations. Catching it in the act only treats the symptom; doing analysis alone does not guarantee no new variations will appear.

Hallucination detection catches false content that has already been produced, either after the fact or in real time; hallucination risk assessment judges in advance how likely a given scenario, model, or data combination is to produce hallucinations. Catching it in the act only treats the symptom; doing analysis alone does not guarantee no new variations will appear.

Fable

The constable and the analyst at the supreme court

Many fake bookkeepers appeared in the capital. They fabricated numbers when doing accounts for merchants—some made mistakes, some deliberately deceived. Constables patrolled the streets and sealed and prosecuted any fake ledgers they found.

They were very diligent in catching them, but the merchants were still uneasy. The fake accountant's methods kept changing: some tampered with ambiguous transactions, and some took advantage of new merchants who did not understand the rules to fish in troubled waters. The constables rushed about as if chasing shadows.

The Court of Judicial Review sent a group of analysts who did not handle specific cases but only reviewed hundreds of past false accounting cases. They found that false accounting occurred most often in three places: when the accounting period was especially tight, when more than three people handled it, and when a new currency was used.

The analyst drew a risk map, marking which account types, time periods, and industries were most prone to trouble. The constables began patrolling by the map, adding staff during tight accounting periods, focusing spot checks on new currency transactions, and prioritizing review of accounts handled by many people.

A new scam appeared: a fake accountant forged transactions in an old currency, which happened to bypass the map. The constables failed to detect it in advance because the map only saw history. The analyst added the new case files overnight, and the patrol routes changed accordingly.

The capital later kept two sets of methods. The constables were responsible for catching false accounts that had already appeared, while analysts were responsible for judging where new false accounts were more likely to emerge. The former reduced errors that had already occurred, while the latter narrowed blind spots; only by feeding discoveries back to each other could they keep up with changing fraud methods.

The prefect also reminded the merchants not to treat the map as a talisman. A map can tell everyone where it is more dangerous, but it cannot audit every ledger on behalf of the constables; constables can catch the errors in front of them, but they cannot say where the next kind of fraud will come from.

Later, every time a new false account was caught, the constable handed the case file to the analyst; every time the map turned red, the analyst also told the constable which street to go to first. The capital relied on these two work loops to connect pursuit after errors appeared with defense before they appeared. Later, when newcomers started, the old masters no longer began with abstract categories, but had them review that failure: where intuition had originally been relied on, why the first patch was insufficient, and how the new arrangement kept risk in a place that could be checked. Only when they could explain these details clearly, and also point out when to continue using it and when to escalate, were they considered to truly understand this approach. Later, they also had newcomers practice this approach: first have newcomers do it the old way once, to see how errors appeared; then redo it according to the new rules, to see which step was blocked in advance and which step left evidence. If newcomers could only recite names, they would still misuse it in similar scenarios; only those who could state the applicable boundaries, costs, and remedies after failure were allowed to take on work independently.

Hallucination detection

Through rules, models, or human review, detect factual errors, logical contradictions, and false content in model outputs in real time or after the fact.

Hallucination risk

Assess the likelihood and severity of hallucinations under specific scenarios, data combinations, model choices, or task types, for prevention and prioritization.

Story Mapping

  • The constable seizes the false accounts: real-time or post-hoc review in hallucination detection
  • The analyst drew a risk map: hallucination risk assessment is an ex ante analysis of scenarios, data and models
  • Extra constables during tight accounting periods: tilting detection resources toward high-risk scenarios
  • New scams bypass the map: risk assessment cannot exhaust all unknown patterns
  • Constable feedback updates the map: detection results continuously feed back into the risk assessment model

Landing it in safe Agent launch

For high-risk scenarios (medical, legal, financial), conduct a hallucination risk assessment first to determine which query types and knowledge domains are most prone to problems, then deploy stronger detection rules and manual spot checks for high-risk domains. After launch, continuously write newly detected hallucination patterns back into the risk assessment.

京ICP备2026014272号