Human in the Loop Is Not the Same as Human in Control
The most common answer to fears about autonomous AI is also, frequently, the least examined
When concerns arise about an AI system operating with too much autonomy, the standard reassurance is that a human remains in the loop. This phrase has become so routine in AI governance discussions that it functions almost as a closing argument, the point at which further scrutiny is assumed to be unnecessary. It deserves considerably more scrutiny than it receives, because a human can be present in a process and still exercise no meaningful control over its outcome.
How the Loop Becomes Hollow
A human reviewing one hundred automated decisions an hour, each requiring a judgement the system’s own confidence score has already suggested is correct, is not exercising the kind of scrutiny the phrase implies. This person is present, technically satisfying the requirement, while functioning in practice as a rubber stamp for whatever the system recommends, because the volume and pacing of the work make genuine independent judgement on each case a practical impossibility.
A human whose approval is required, but whose disapproval carries a documented cost, a delay flagged in a performance review, a productivity metric that quietly penalises the reviewer who slows the process down, faces structural pressure toward approval regardless of the merits of a given case. The human is in the loop. The incentive structure has already decided which way the loop bends.
A human who lacks the specific expertise or the specific information needed to actually evaluate the system’s recommendation is present without being equipped to exercise the oversight their presence is meant to represent. A reviewer asked to approve a complex financial or medical recommendation without the training to independently assess it is providing the appearance of human judgement without its substance.
Why This Matters More as Systems Grow More Capable
The temptation to treat human oversight as satisfied by mere presence grows precisely as systems become more capable and more confident, which is exactly backward. A less capable system that frequently produces obviously wrong output invites genuine scrutiny, because the errors are visible and the reviewer’s attention is naturally engaged. A highly capable system that is right the overwhelming majority of the time produces the opposite effect: reviewers habituate to its accuracy, trust accumulates, and the rare case where the system is confidently wrong is the one most likely to pass through a human loop that has learned, correctly on average but catastrophically in this instance, to expect the system to be right.
This is not a reason to avoid capable systems. It is a reason to design oversight around the specific failure mode capable systems actually produce, rather than around a general assurance that a human is somewhere in the process.
What Genuine Human Control Requires
Time and workload calibrated to allow actual scrutiny, not merely nominal review, which means fewer decisions reviewed more carefully rather than every decision reviewed too quickly to matter. Incentive structures that do not quietly penalise the reviewer who disagrees with the system, since a reviewer who pays a professional cost for disagreement will learn, rationally, to agree. Sufficient expertise and information for the reviewer to form an independent judgement rather than merely ratify the system’s confidence. And genuine authority to override, including the practical ability to do so without the override itself becoming an event requiring justification disproportionate to a routine approval.
Catholic moral tradition insists that responsibility requires the actual capacity to have acted otherwise, not merely a formal role that implies such capacity existed. A human nominally in the loop who could not, in any realistic sense, have exercised independent judgement has not been given moral responsibility for the outcome. They have been given the appearance of it, which is considerably more dangerous than no human involvement at all, because it creates false confidence that oversight occurred where none functionally did.
The question worth asking of any system described as having human oversight is not whether a person is present. It is whether that person had the time, the incentive, the expertise, and the authority to actually disagree, and would have faced no unreasonable cost for doing so.