Hallucination is a Feature, Not a Bug
Engine Room Article 10: Understanding Why AI Makes Things Up
How Text Generation Works
Language models are next-token predictors. Given a sequence of text, they generate the most likely continuation based on patterns learned from training data.
Here’s the important part: ‘most likely’ doesn’t mean ‘true.’ The model doesn’t have a concept of truth - it has a concept of plausibility.
The model doesn’t have a concept of truth - it has a concept of plausibility. Hallucination isn’t a malfunction; it’s inherent to probabilistic text generation.
The gap between how hallucination is discussed in demos (‘rare edge case we’re fixing’) versus how it appears in deployment (‘regular occurrence requiring mitigation’) suggests the framing itself might be incomplete.
Why It Can’t Be Eliminated
Hallucination is inherent to probabilistic text generation. Mitigation strategies can reduce frequency but not eliminate it.
Current approaches - retrieval augmentation, chain-of-thought prompting, output verification - all reduce hallucination rates. None eliminate the underlying property that makes hallucination possible.
The Governance Implication
If hallucination can’t be eliminated technically, it becomes a governance problem: how do you manage systems that will sometimes confidently produce incorrect outputs?
Instead of ‘how do we fix hallucination?’ the better question might be ‘for which use cases is this error rate acceptable?’
Hallucination is a property of probabilistic text generation, not a bug being fixed. Governance is about managing that property, not waiting for it to disappear.
Related: 07-source—engine-room-series