The next cyber security challenge: can we trust the AI agents we deploy?
For years, cyber security has been built around a simple assumption: humans make decisions and systems execute them.
Agentic AI challenges that assumption.
As organisations move beyond experimentation and begin deploying AI agents capable of analysing information, making recommendations and triggering actions, a new cyber security challenge is emerging. The question is no longer just whether organisations can defend themselves against AI-enabled attacks. It is whether they can trust the AI systems they are introducing into their own operations.
Much of the discussion around artificial intelligence (AI) and cyber security has focused on how threat actors are using AI to increase the speed, scale and sophistication of attacks. The UK's National Cyber Security Centre (NCSC) and its Five Eyes partners have warned that AI is lowering barriers to entry for cyber criminals and accelerating the exploitation of vulnerabilities.
These risks are real. But as organisations increasingly deploy AI to support business processes, customer interactions, software development and operational decision-making, another question is coming into sharper focus:
How do we secure the AI agents we deploy?
The challenge goes beyond the model
When discussing AI security, there is often a tendency to focus on the model itself. In reality, AI agents rely on a much broader ecosystem of data sources, applications, APIs, cloud services and third-party integrations. Connections create business value, but also potential risk. As AI becomes more autonomous, the attack surface expands.
An agent may have access to sensitive information, customer records, internal systems or operational processes. It may also take actions at a speed and scale that make continuous human oversight difficult. This creates new challenges, from prompt injection and data manipulation to agent-to-agent exploitation, unintended actions and the amplification of errors across multiple systems.
The conversation is no longer simply about whether an AI model is accurate or effective. It’s about whether organisations can continuously trust the environment around it.
AI is accelerating familiar risks
One misconception is that AI is creating entirely new cyber threats. More often, it’s accelerating existing ones.
Phishing remains one of the most effective attack techniques available to cyber criminals. OurState of Cybersecurity 2025 report found that phishing was implicated in 59.9% of security incidents observed across customer networks during 2024. Generative AI is making phishing campaigns more convincing, more personalised, and easier to scale. Industry research highlights the pace of this shift, with 82.6% of phishing emails analysed in 2025 found to utilise AI, according to KnowBe4’s latest threat intelligence report. AI is also accelerating vulnerability discovery, reconnaissance and social engineering activities.
The result is not necessarily a different type of threat, but a more efficient and scalable version of threats organisations already face.
The same lesson applies to organisations deploying AI agents. If attackers are becoming more effective through automation, organisations must ensure that governance, assurance and security controls can operate at a comparable pace.
Trust can’t be assumed
Historically, organisations have relied on governance frameworks to manage risk. People are trained, processes are documented, approvals are recorded and responsibilities are assigned.
Agentic AI introduces new questions.
What information should an AI agent be allowed to access?
What decisions should it be allowed to make?
What level of autonomy is appropriate?
How do organisations know when an AI system behaves unexpectedly?
Most importantly, who remains accountable?
These are not simply technical questions. They are leadership questions. The NCSC's Board Toolkit states that cyber security is "not just good IT" and should be integrated into organisational risk management and decision-making. Organisations need to think carefully about where meaningful human oversight should remain, particularly in areas where AI-driven decisions could have significant operational, financial or societal consequences.
What we're seeing across UK organisations
Across our work with government departments, defence organisations and other highly regulated industries, we're seeing a noticeable shift in how leaders think about cyber security and AI. The conversation has moved beyond whether organisations should adopt AI. Increasingly, leaders want to understand how they can adopt it safely, demonstrate appropriate oversight and build trust in systems that may be making decisions at greater speed and scale than humans can realistically monitor.
We're also seeing growing interest in governance, accountability and assurance. Boards are asking how AI decisions can be explained, how AI systems can be monitored and where responsibility sits when autonomous systems make recommendations or take actions. These are not questions that can be answered by technology alone. They require leadership, clear governance and a strong resilience mindset.
The organisations making the greatest progress are treating AI as a business transformation issue, not simply a technology initiative. They recognise that building trust will be just as important as deploying capability.
Resilience will become the competitive advantage
The good news is that organisations don’t need an entirely new security playbook. The fundamentals of good cyber security remain consistent: secure-by-design approaches, strong identity and access management, effective supply chain assurance, continuous monitoring and clear accountability.
However, these foundations must now be applied to a world where AI systems can act with increasing autonomy. Success will depend on finding the right balance between innovation and control, enabling organisations to benefit from agentic AI while retaining confidence in how decisions are made and actions are executed.
The organisations that succeed in the age of agentic AI won’t necessarily be those that move faster, they will be those that build trust.
This thinking also aligns with broader UK cyber resilience initiatives. The UK Government's Cyber Resilience Pledge encourages organisations to make cyber a board responsibility, engage with NCSC services and adopt a risk-based approach to resilience across supply chains.
As AI becomes embedded into day-to-day operations, cyber security's role is expanding. Organisations must continue to defend against AI-enabled threats, while also ensuring that the AI agents they deploy are secure, governed and operating within clearly defined boundaries. That means moving beyond traditional security controls and adopting approaches that continuously monitor, test and assure AI systems, while maintaining meaningful human oversight where decisions carry significant operational, financial or societal impact. The challenge is no longer simply protecting organisations from AI. It is securing AI itself as it becomes part of the organisation.
Ultimately, the organisations that realise the greatest value from agentic AI will be those that build trust into deployment from the outset. In the age of autonomous systems, resilience, accountability and oversight will become as important as capability and speed. The future of AI will not be defined by how autonomous agents become, but by how confidently organisations can deploy them, govern them and trust them to act safely at scale.
