Executive summary
Hallucination is not simply any unexpected wording. The critical failures are claims, citations, or conclusions that are unsupported by approved evidence or conflict with known constraints.
Method
Risk-based testing framework combining source verification, controlled datasets, adversarial prompts, abstention checks, and human escalation.
What the evidence suggests
Separate failure categories
Unsupported generation, incorrect retrieval, stale evidence, citation mismatch, policy breach, and normal wording variation need different fixes.
Ground truth has boundaries
Teams must define authoritative sources, version dates, permitted external knowledge, and cases where the system should abstain.
One percentage is insufficient
Report severity, claim type, evidence coverage, task slice, and business impact alongside an aggregate rate.
Practical test protocol
- 01Define authoritative evidence and acceptable uncertainty.
- 02Create supported, unsupported, conflicting, stale, and unanswerable cases.
- 03Verify every material factual claim and citation.
- 04Test refusal and calibrated uncertainty.
- 05Repeat cases and compare model or prompt versions.
- 06Route high-impact failures to owners and qualified human reviewers.
Control hallucination through evidence boundaries, measurable failure taxonomies, grounded workflows, and risk-based escalation—not a promise of zero errors.
Limitations
No finite test suite can prove that a generative system will never produce an unsupported answer. Controls must continue in production.