What If Hallucination Isn't the Risk You Should Be Managing?

The obsession with catching AI's mistakes may be distracting from a deeper question about what we've handed it

Every serious conversation about AI adoption eventually arrives at the same checklist item: mitigate hallucination risk. Build verification layers. Train staff to fact-check outputs. Flag low-confidence answers. All of this is sensible, and none of it is wrong. But there is a quieter question hiding behind the risk-management language, one that no verification layer can answer: why did a system with no understanding of truth become the first draft of so many human decisions in the first place?

A Category Error Dressed as a Technical Problem

Hallucination gets treated as a bug to be minimized, the way a manufacturer minimizes defect rates. That framing assumes the underlying activity is sound and only needs tighter tolerances. But a model does not hallucinate the way a person misremembers a fact. It generates plausible-sounding text without any internal relationship to whether that text is true, and it does this every time, not only on the occasions we happen to catch it. What we call a hallucination is simply the moment the plausible and the true came apart visibly enough for someone to notice. The system was never doing anything different in the cases we didn't catch.

This is not a call to abandon the tools. It is a call to stop describing the problem in terms that make it sound solvable by better engineering alone, when what it actually requires is a permanent posture of human responsibility for every claim these systems produce.

Truth Was Never a Technical Property

Catholic thought has always insisted that truth is not merely correspondence between a statement and a fact to be checked off a list. Truth is something a person is responsible for, something spoken in relationship, something that carries moral weight because a human being stands behind it. A system that produces confident, well-formed sentences with no capacity to be responsible for them is not a flawed truth-teller. It is not a truth-teller at all. It is a very sophisticated pattern-completion engine that happens, often, to produce sentences that are also true.

Treating hallucination purely as a risk to be managed keeps the conversation inside a frame where the goal is an acceptable error rate. But no error rate, however low, changes the fact that the system itself has no stake in truth, no conscience to violate, and no capacity for the kind of accountability that a human speaker owes to a listener. The risk-management frame quietly transfers moral weight onto the machine's output instead of keeping it where it has always belonged: on the person who chose to rely on that output without verifying it.

What Actually Needs Governing

The practical shift this suggests is small but important. Instead of asking only "how do we catch the errors," organizations and individuals might ask "where have we let a system's fluency substitute for a person's judgment." The two questions produce different behavior. The first produces better filters. The second produces a culture where a human being remains answerable, by name, for every consequential claim that reaches a patient, a parishioner, a student, or a customer, regardless of which tool helped draft it.

This is not a technical fix. It is a restoration of an old and simple principle: a machine can assist the search for truth, but it cannot bear responsibility for it. Only a person can do that, and no amount of hallucination-reduction engineering should be allowed to obscure who that person is.