How to reduce LLM hallucinations.

Step 1: Define what a hallucination is for you

"Hallucination" is a fuzzy word. Break it down into failure classes that your application actually cares about: fabricated facts (invented numbers, dates, names), wrong citations (real sources, wrong content), unsupported inferences (technically true but not derivable from context), and format failures (wrong schema, missed refusal). Write these down. Every downstream metric should map to one of them.

Step 2: Build an evaluation harness before you tune anything

50 to 200 questions with expected answers or acceptance criteria. Score with an LLM as judge for scale, plus a 10% human spot check to keep the judge honest. Include: happy path questions, edge cases, questions with no answer in your data (should refuse), and adversarial questions designed to trigger fabrication. Run it against your current system. That's your baseline.

Step 3: Fix retrieval first, model second

The single largest source of hallucinations we see in enterprise pilots is bad retrieval. The chunk that contained the answer wasn't in the top k. Fix that before you touch the model. Tactical wins: smarter chunking that respects semantic boundaries, hybrid retrieval (BM25 + vector), a re ranker on top 100 results, and metadata filters that constrain retrieval to the right document class.

Step 4: Constrain the prompt to refuse when context is thin

The default LLM behaviour is to attempt an answer. Override it. Explicit instructions like "if the provided context does not support an answer, respond 'I don't know based on the available information'", plus a few in prompt refusal examples, radically improve behaviour on unanswerable questions. Combine with a confidence threshold on retrieval scores: if nothing scored well, refuse.

Step 5: Add a knowledge graph for multi hop and structured facts

If your questions require walking relationships (customer to plan to feature to ticket, or patient to condition to medication to interaction) no amount of vector tuning fixes it. This is when GraphRAG pays for itself. The graph returns the actual entities and edges the answer depends on; the LLM composes them into prose. Hallucination on these question types typically drops from double digits to near zero. We covered the architecture on the GraphRAG implementation page.

Step 6: Require citations in every answer, then verify them

Force the model to output source references (document ID + chunk offset, or graph node ID) with every fact. Post generate, verify that each cited source (a) exists and (b) actually supports the claim (a second LLM pass, or a smaller purpose trained classifier). Reject answers that fail. The user experience is "asking again"; the alternative is quietly shipping fabrications.

Step 7: Re run the evaluation harness on every change

Treat the hallucination rate the way you treat latency or cost: a first class metric on every deploy. When it regresses, roll back and diagnose. Over 3 to 6 months of this discipline, systems that started at 15 to 25% hallucination on domain questions typically land in the 1 to 3% band, with the remaining errors concentrated in genuinely ambiguous cases where a human would disagree too.

Two anti patterns to avoid

• "Just switch models." The gain from a stronger base model is usually a fraction of the gain from fixed retrieval and refusal shaped prompts. Model swaps are cheap to try, but they rarely fix a retrieval problem.
• "Fine tuning will fix it." Fine tuning teaches style. It rarely teaches facts, and teaching facts is exactly where fine tunes go wrong (they overfit to the training set and confidently confabulate outside it). Ground in retrieval; don't bake in facts.