The Gap Between AI Interest and AI Impact
Most organisations can point to AI experiments. Far fewer can point to AI that changed anything that actually mattered
Interest in AI is now nearly universal among organisations of every size, and pilot projects have proliferated accordingly: a chatbot trial here, a forecasting experiment there, a generative tool tested by one team and quietly abandoned six months later. What is far rarer is AI that moved past the pilot stage into something that measurably changed how an organisation serves the people who depend on it. The gap between interest and impact deserves more attention than either extreme of the current conversation, the breathless promotional case and the dismissive backlash, tends to give it.
Where Most Efforts Stall
The most common failure point is not technical. It is the absence of a clearly defined problem before the technology is selected. A team excited about a new capability will frequently look for a use case to justify the enthusiasm, rather than starting from a genuine organisational problem and asking honestly whether AI is the right tool to address it. Projects that start this way tend to produce technically functional pilots that never advance, because nobody can articulate specifically what business or human problem success would actually solve.
A second common failure is treating a successful pilot as though it will scale without further work. A tool that performs well when a small, engaged, technically comfortable team is testing it frequently behaves quite differently once it reaches a broader population with less patience for imperfection and less context for what the tool is trying to do. The gap between pilot performance and production performance is where a great many promising projects quietly die, usually without anyone conducting an honest post-mortem on why.
A third failure is measuring the wrong thing throughout. Organisations frequently report AI adoption in terms of usage, how many people tried the tool, how many queries it processed, rather than in terms of outcomes, whether the people it was meant to help are actually better served. Usage is easy to measure and impact is harder, and the gap between the two metrics is where organisations convince themselves a project succeeded when the honest answer is closer to uncertain.
What Moving From Interest to Impact Actually Requires
Starting from a specific, named problem serious enough that solving it would matter to someone beyond the team excited about the technology. If the answer to “what problem does this solve” is vague or internally focused, the project is not yet ready to begin.
Testing at a scale and diversity close enough to real deployment conditions that the pilot’s success is a genuine signal rather than an artefact of a friendly test environment. This is slower and less exciting than a quick internal demo, and it is the only way to know whether a promising pilot will actually survive contact with ordinary use.
Measuring outcomes that matter to the people the project claims to serve, defined and agreed before the project begins rather than selected afterward from whatever numbers happen to look favourable. If an AI tool is meant to help customers, the outcome measure should be something a customer would recognise as an improvement, not merely an internal efficiency statistic.
Why the Discipline Matters Beyond the Balance Sheet
Catholic social teaching holds that resources devoted to any enterprise carry an obligation to serve genuine human need, and an organisation that spends significant time and money on AI experiments that never produce real benefit for anyone has not merely wasted a budget line. It has diverted attention and resources that could have gone toward efforts that actually served the people the organisation exists to serve.
The measure of a mature AI programme is not the number of experiments it has run. It is the number of experiments that moved all the way from genuine curiosity to a measurable difference in someone’s actual life, and the honesty to say plainly, of everything that did not make that journey, what was learned and why it was set aside.