Why Agentic AI is Eroding the Safeguard Everyone’s Counting On
The New Nothing
People have fallen in love with Generically Plausible Text. Twenty-page responses to simple questions, written by people who six months ago might have produced nothing at all. The capability of AI is so flabbergasting that organisations end up blindsided by its limitations. The better the models get, the wider the gap. For many knowledge workers, AI has become the new nothing: an overload of output that hides the absence of thought. And I feel it too. As a data scientist, the pull to skip straight to a working prototype is real, and every time I give in, I understand the thing I built a little less.
Human-Demoted-From-The-Loop (HDFTL)
“Human judgement reviews the output”.
“It’s not that long, it’s bulleted”.
Leaders cite human judgement as the final line of defence. It is the protected layer everybody gestures at as the safeguard against agentic AI failure.
But judgement is not a free-standing capacity. It comes from having done the work. The reviewer also has to know what they're doing in the first place, or the Generically Plausible Text reads as competent because nothing in their experience tells them otherwise. Through the struggle of debugging, redrafting, and iterating, we build a working theory of how the thing behaves, and from that, judgement. Automate that struggle away with AI agents, and the very thing protecting us against the failures of AI is being eroded by the deployment itself. The safeguard is eating itself.
Right now, companies feel threatened.
“Adopt agentic AI or fall behind.”
“Companies that aren’t AI-native in five years will die.”
The existential framing is being pushed hardest by the people selling the models and the first movers betting on the spread. As someone who’s spent a career inside AI and ML, the view from the practitioner side looks different. The tech is real. The panic is marketing.
But if we don’t do it right, we don’t evolve. We atrophy, because we skip the cognitive process that produces the judgement we then claim is the human edge.
The empirical picture is harder still. Recent studies suggest that the human review layer doesn’t work the way we want it to (Bedard et al., 2026). It isn’t how the mind operates. We are not equipped to apply rigorous scrutiny to fluent, confident outputs that already look correct, especially under time pressure and when the social cost of disagreeing with the machine compounds across a long working day. And the volume makes it worse. When there’s twenty pages to check and ten more to go, approving and hoping beats interrogating every line.
To a coder, this looks different. You can see what’s missing because you know how it gets built.
The Erosion is Already Underway
I find myself returning to Peter Naur’s 1985 essay on programming as theory-building (Naur, 1985). The argument is simple: software is not the code, it is the theory of the system that lives in the heads of people who built it. The theory is downstream of doing. Judgement is downstream of the theory.
Automate the doing and the theory never forms. The gap between what AI produces and what its users can meaningfully evaluate, the calibration paradox, is widening because users are skipping the very experience that would let them evaluate it.
The pattern is visible already.
VirtusLab’s 2026 post-mortem across their client engagements puts it sharply (VirtusLab, 2026):
"You get code that might be identical to what you'd have come up with after two days. But behind your code would stand a model: what other solutions exist and why they're worse, where the limits of this approach are, and what will break when requirements change. That understanding doesn't live in the code. It lives in the head of the person who went through the process. Behind AI's code, nothing stands."
The effect is also uneven, which is telling. At Adidas, teams with loosely coupled architectures saw productivity gains of 20-30% from AI coding tools, while teams with tightly coupled systems, where understanding the why of the architecture is non-negotiable, saw no benefit at all (Kim, 2025). Where theory matters most is precisely where AI assistance gives the least, because there is no theory available to transfer.
A sharp reader will counter that this is the autopilot story all over again. Aviation automated the syntactic work of flying and a generation of doomsdayers said pilot intuition would die. Training adapted instead. A fair point and also exactly the problem. Aviation rebuilt the training paradigm before scaling the automation, and it did that because the failures were too terrifying to ignore (planes falling out of the sky). Agentic AI fails quietly. Consider a 0.02% drop in accuracy buried in a B2B process no one is really watching that closely. Where’s the wreckage to force the rebuild? We’re scaling faster than we’re designing an equivalent, and nothing is going to make us stop.
What UK Businesses Should Actually Do
Pair agentic AI with deliberate theory-preserving practices. Keep humans in the design phase, not just the review phase. Use AI to widen exploration, not to skip the thinking. Measure where intuition is forming, not just throughput. If your reviewers stop being able to articulate why they reject an agent’s output, the theory has eroded, regardless of what the velocity numbers say.
This is also where the line between “Enterprise AI” and “Chatbot AI” matters. Chatbot AI is one prompt and an answer. The user trusts the surface because there is no governance behind it, no evaluation harness underneath, and no theory of the domain to fall back on when the surface is wrong. Enterprise AI is the discipline of pairing the model with the right context, the right data, the right guardrails, and the practitioners whose theory of the domain is doing the load-bearing work. The model is often the same. The system around it is what determines whether the deployment compounds capability or quietly erodes it. At Braidr, this is the discipline we hold ourselves to - design the system that keeps the theory load-bearing, then let the model do its work inside it.
“We saw this all before. We lived through the dot-com bubble.”
Yes, every senior generation has thought the next was missing the real skill, and they have a point. But previous abstractions were syntactic: faster ways to express the same thinking. Agentic AI is cognitive. It does the thinking. That is a real distinction, and it is why this transition deserves more care than the last few.
So I've changed how I work. Of course, I still reach for the prototype, but I make myself build the theory before I even try to defend the output, because the version of me that skipped that step understood less every time. That's the whole discipline. Do the work, then scale it. Keep building the theory, or the safeguard you're counting on quietly disappears.
Bedard, J., Kropp, M., Hsu, M., Karaman, O., Hawes, J. and Kellerman, G.R. (2026) 'When using AI leads to "brain fry"', Harvard Business Review [Online]. Available at: https://hbr.org/2026/03/when-using-ai-leads-to-brain-fry [Accessed: 26 May 2026].
VirtusLab (2026) 'Cognitive debt: The code nobody understands', VirtusLab Blog [Online]. Available at: https://virtuslab.com/blog/ai/cognitive-debt-the-code-nobody-understands (Accessed: 26 May 2026).
Kim, G. (2025) 'Foreword', in DeBellis, D. et al. DORA state of AI-assisted software development 2025. Google Cloud, pp. 8–10. Available at: https://dora.dev/research (Accessed: 26 May 2026).
Naur, P., 1985. Programming as theory building. Microprocessing and Microprogramming, 15(5), pp.227-240.
James Wolman
Head of Data Science, Braidr
James Wolman
Head of Data Science, Braidr
techUK - Seizing the AI Opportunity
The UK is a global leader in AI innovation, development and adoption.
AI has the potential to boost UK GDP by £550 billion by 2035, making adoption an urgent economic priority. techUK and our members are committed to working with the Government to turn the AI Opportunities Action Plan into reality. Together we can ensure the UK seizes the opportunities presented by AI technology and continues to be a world leader in AI development.
Get involved: techUK runs a busy calendar of activities including events, reports, and insights to demonstrate some of the most significant AI opportunities for the UK. Our AI Hub is where you will find details of all upcoming activity. We also send a monthly AI newsletter which you can subscribe to here.
Read Okta's guest blog, discussing how the core opportunity for British organisations today is enabling teams to safely move AI agents into production at scale and at speed.
How are enterprise AI agents reshaping UK organisations? NIX United shares a real-world device management case and what responsible agentic AI adoption actually takes.
Across UK sectors, the opportunities are clear: faster decision-making, more personalised services and reduced friction for consumers. But the real question is not simply where agentic AI can be used. It is whether it will be trusted enough to be used at scale. Read Jocelyn Paulley's piece to find out more.
Subscribe to our AI newsletter
AI and Data Analytics updates
Sign-up to our monthly newsletter to get the latest updates and opportunities from our AI and Data Analytics Programme straight to your inbox.
Contact the team
Kir Nuthi
Head of AI and Data, techUK
Kir Nuthi
Head of AI and Data, techUK
Kir Nuthi is the Head of AI and Data at techUK.
She holds over seven years of Government Affairs and Tech Policy experience in the US and UK. Kir previously headed up the regulatory portfolio at a UK advocacy group for tech startups and held various public affairs in US tech policy. All involved policy research and campaigns on competition, artificial intelligence, access to data, and pro-innovation regulation.
Kir has an MSc in International Public Policy from University College London and a BA in both Political Science (International Relations) and Economics from the University of California San Diego.
Outside of techUK, you are likely to find her attempting studies at art galleries, attempting an elusive headstand at yoga, mending and binding books, or chasing her dog Maya around South London's many parks.
Usman joined techUK in January 2024 as Programme Manager for Artificial Intelligence.
He leads techUK’s AI Adoption programme, supporting members of all sizes and sectors in adopting AI at scale. His work involves identifying barriers to adoption, exploring solutions, and helping to unlock AI’s transformative potential, particularly its benefits for people, the economy, society, and the planet. He is also committed to advancing the UK’s AI sector and ensuring the UK remains a global leader in AI by working closely with techUK members, the UK Government, regulators, and devolved and local authorities.
Since joining techUK, Usman has delivered a regular drumbeat of activity to engage members and advance techUK's AI programme. This has included two campaign weeks, the creation of the AI Adoption Hub (now the AI Hub), the AI Leader's Event Series, the Putting AI into Action webinar series and the Industrial AI sprint campaign.
Before joining techUK, Usman worked as a policy, regulatory and government/public affairs professional in the advertising sector. He has also worked in sales, marketing, and FinTech.
Usman holds an MSc from the London School of Economics and Political Science (LSE), a GDL and LLB from BPP Law School, and a BA from Queen Mary University of London.
When he isn’t working, Usman enjoys spending time with his family and friends. He also has a keen interest in running, reading and travelling.
Sue leads techUK's Technology and Innovation work. This includes work programmes on AI, Cloud, Data, Quantum, Semiconductors, Digital ID and Digital ethics as well as emerging and transformative technologies and innovation policy. In 2025, Sue was honoured with an Order of the British Empire (OBE) for services to the Technology Industry in the New Year Honours List. She has also been recognised as one of the most influential people in UK tech by Computer Weekly's UKtech50 Longlist and was inducted into the Computer Weekly Most Influential Women in UK Tech Hall of Fame.
A key influencer in driving forward the tech agenda in the UK, in December 2025 Sue was appointed to the UK Government’s Women in Tech Taskforce by the Technology Secretary of State. She also sits on the UK Government’s Smart Data Council, Satellite Applications Catapult Advisory Group, Bank of England’s AI Consortium and BSI’s Digital Strategic Advisory Group. Previously, Sue was a member of the Independent Future of Compute Review and co-chaired the National Data Strategy Forum. As well as being recognised in the UK's Big Data 100 and the Global Top 100 Data Visionaries in 2020, Sue has been shortlisted for the Milton Keynes Women Leaders Awards and has been a judge for the Loebner Prize in AI, the UK Tech 50 and annual UK Cloud Awards. She is a regular industry speaker on issues including AI ethics, data protection and cyber security.
Prior to joining techUK in January 2015, Sue was responsible for Symantec's Government Relations in the UK and Ireland. Before that, Sue was senior policy advisor at the Confederation of British Industry (CBI). Sue has an BA degree on History and American Studies from Leeds University and a Master’s Degree in International Relations and Diplomacy from the University of Birmingham. Sue is a keen sportswoman and in 2016 achieved a lifelong ambition to swim the English Channel.
Visit our AI Hub - the home of all our AI content:
Enquire about membership:
Become a techUK member
Our members develop strong networks, build meaningful partnerships and grow their businesses as we all work together to create a thriving environment where industry, government and stakeholders come together to realise the positive outcomes tech can deliver.
James is Head of Data Science at Braidr, a London-based data science and AI consultancy helping brands mature their data and AI capabilities through practical, ROI-focused implementations.
