The next cyber security challenge: can we trust the AI agents we deploy?

For years, cyber security has been built around a simple assumption: humans make decisions and systems execute them.

As organisations move beyond experimentation and begin deploying AI agents capable of analysing information, making recommendations and triggering actions, a new cyber security challenge is emerging. The question is no longer just whether organisations can defend themselves against AI-enabled attacks. It is whether they can trust the AI systems they are introducing into their own operations.

Much of the discussion around artificial intelligence (AI) and cyber security has focused on how threat actors are using AI to increase the speed, scale and sophistication of attacks. The UK's National Cyber Security Centre (NCSC) and its Five Eyes partners have warned that AI is lowering barriers to entry for cyber criminals and accelerating the exploitation of vulnerabilities.

These risks are real. But as organisations increasingly deploy AI to support business processes, customer interactions, software development and operational decision-making, another question is coming into sharper focus:

How do we secure the AI agents we deploy?

The challenge goes beyond the model

When discussing AI security, there is often a tendency to focus on the model itself. In reality, AI agents rely on a much broader ecosystem of data sources, applications, APIs, cloud services and third-party integrations. Connections create business value, but also potential risk. As AI becomes more autonomous, the attack surface expands.

An agent may have access to sensitive information, customer records, internal systems or operational processes. It may also take actions at a speed and scale that make continuous human oversight difficult. This creates new challenges, from prompt injection and data manipulation to agent-to-agent exploitation, unintended actions and the amplification of errors across multiple systems.

The conversation is no longer simply about whether an AI model is accurate or effective. It’s about whether organisations can continuously trust the environment around it.

AI is accelerating familiar risks

One misconception is that AI is creating entirely new cyber threats. More often, it’s accelerating existing ones.

Phishing remains one of the most effective attack techniques available to cyber criminals. OurState of Cybersecurity 2025 report found that phishing was implicated in 59.9% of security incidents observed across customer networks during 2024. Generative AI is making phishing campaigns more convincing, more personalised, and easier to scale. Industry research highlights the pace of this shift, with 82.6% of phishing emails analysed in 2025 found to utilise AI, according to KnowBe4’s latest threat intelligence report. AI is also accelerating vulnerability discovery, reconnaissance and social engineering activities.

The result is not necessarily a different type of threat, but a more efficient and scalable version of threats organisations already face.

The same lesson applies to organisations deploying AI agents. If attackers are becoming more effective through automation, organisations must ensure that governance, assurance and security controls can operate at a comparable pace.

Trust can’t be assumed

Historically, organisations have relied on governance frameworks to manage risk. People are trained, processes are documented, approvals are recorded and responsibilities are assigned.

These are not simply technical questions. They are leadership questions. The NCSC's Board Toolkit states that cyber security is "not just good IT" and should be integrated into organisational risk management and decision-making. Organisations need to think carefully about where meaningful human oversight should remain, particularly in areas where AI-driven decisions could have significant operational, financial or societal consequences.

What we're seeing across UK organisations

Across our work with government departments, defence organisations and other highly regulated industries, we're seeing a noticeable shift in how leaders think about cyber security and AI. The conversation has moved beyond whether organisations should adopt AI. Increasingly, leaders want to understand how they can adopt it safely, demonstrate appropriate oversight and build trust in systems that may be making decisions at greater speed and scale than humans can realistically monitor.

We're also seeing growing interest in governance, accountability and assurance. Boards are asking how AI decisions can be explained, how AI systems can be monitored and where responsibility sits when autonomous systems make recommendations or take actions. These are not questions that can be answered by technology alone. They require leadership, clear governance and a strong resilience mindset.

The organisations making the greatest progress are treating AI as a business transformation issue, not simply a technology initiative. They recognise that building trust will be just as important as deploying capability.

Resilience will become the competitive advantage

The good news is that organisations don’t need an entirely new security playbook. The fundamentals of good cyber security remain consistent: secure-by-design approaches, strong identity and access management, effective supply chain assurance, continuous monitoring and clear accountability.

However, these foundations must now be applied to a world where AI systems can act with increasing autonomy. Success will depend on finding the right balance between innovation and control, enabling organisations to benefit from agentic AI while retaining confidence in how decisions are made and actions are executed.

The organisations that succeed in the age of agentic AI won’t necessarily be those that move faster, they will be those that build trust.

This thinking also aligns with broader UK cyber resilience initiatives. The UK Government's Cyber Resilience Pledge encourages organisations to make cyber a board responsibility, engage with NCSC services and adopt a risk-based approach to resilience across supply chains.

As AI becomes embedded into day-to-day operations, cyber security's role is expanding. Organisations must continue to defend against AI-enabled threats, while also ensuring that the AI agents they deploy are secure, governed and operating within clearly defined boundaries. That means moving beyond traditional security controls and adopting approaches that continuously monitor, test and assure AI systems, while maintaining meaningful human oversight where decisions carry significant operational, financial or societal impact. The challenge is no longer simply protecting organisations from AI. It is securing AI itself as it becomes part of the organisation.

Ultimately, the organisations that realise the greatest value from agentic AI will be those that build trust into deployment from the outset. In the age of autonomous systems, resilience, accountability and oversight will become as important as capability and speed. The future of AI will not be defined by how autonomous agents become, but by how confidently organisations can deploy them, govern them and trust them to act safely at scale.

Cyber Resilience Programme activities

techUK brings together key players across the cyber security sector to promote leading-edge UK capabilities, build networks and grow the sector. techUK members have the opportunity to network, share ideas and collaborate, enabling the industry as a whole to address common challenges and opportunities together. Visit the programme page here.

Upcoming events

Latest news and insights

Learn more and get involved

Cyber Resilience updates

Sign-up to get the latest updates and opportunities from our Cyber Resilience programme.

Meet the team

Jill Broom

Head of Cyber Resilience, techUK

Jill leads the techUK Cyber Resilience programme, having originally joined techUK in October 2020 as a Programme Manager for the Cyber and Central Government programmes. She is responsible for managing techUK's work across the cyber security ecosystem, bringing industry together with key stakeholders across the public and private sectors. Jill also provides the industry secretariat for the Cyber Growth Partnership, the industry and government conduit for supporting the growth of the sector. A key focus of her work is to strengthen the public–private partnership across cyber to support further development of UK cyber security and resilience policy.

Before joining techUK, Jill worked as a Senior Caseworker for an MP, advocating for local communities, businesses and individuals, so she is particularly committed to techUK’s vision of harnessing the power of technology to improve people’s lives. Jill is also an experienced editorial professional and has delivered copyediting and writing services for public-body and SME clients as well as publishers.

Email:: [email protected]
Website:: www.techuk.org/
LinkedIn:: https://www.linkedin.com/in/jill-broom-19aa824

Read lessmore

Annie Collings

Senior Programme Manager, Cyber Resilience, techUK

Annie is the Programme Manager for Cyber Resilience at techUK. She first joined as the Programme Manager for Cyber Security and Central Government in September 2023.

In her role, Annie supports the Cyber Security SME Forum, engaging regularly with key government and industry stakeholders to advance the growth and development of SMEs in the cyber sector. Annie also coordinates events, engages with policy makers and represents techUK at a number of cyber security events.

Before joining techUK, Annie was an Account Manager at a specialist healthcare agency, where she provided public affairs support to a wide range of medical technology clients. She also gained experience as an intern in both an MP’s constituency office and with the Association of Independent Professionals and the Self-Employed. Annie holds a degree in International Relations from Nottingham Trent University.

Email:: [email protected]
Twitter:: anniecollings24
LinkedIn:: https://www.linkedin.com/in/annie-collings-270150158/

Read lessmore

Olivia Staples

Junior Programme Manager - Cyber Resilience, techUK

Olivia Staples joined techUK in May 2025 as a Junior Programme Manager in the Cyber Resilience team.

She supports the programs mission to promote cyber resilience by engaging key commercial and government stakeholders to shape the cyber resilience policy towards increased security and industry growth. Olivia assists in member engagement, event facilitation and communications support.

Before joining techUK, Olivia gained experience in research, advocacy, and strategic communications across several international organisations. At the Munich Security Conference, she supported stakeholder engagement and contributed to strategic communications. She also worked closely with local and national government stakeholders in Spain and Italy, where she was involved in policy monitoring and advocacy for both public and private sector clients.

Olivia holds an MSc in Political Science (Comparative Politics and Conflict Studies) from the London School of Economics (LSE) and a BA in Spanish and Latin American Studies from University College London (UCL).

Outside of tech, Olivia enjoys volunteering with local charities and learning Norwegian.

Email:: [email protected]

Read lessmore

The next cyber security challenge: can we trust the AI agents we deploy?

How do we secure the AI agents we deploy?

The challenge goes beyond the model

AI is accelerating familiar risks

Trust can’t be assumed

What we're seeing across UK organisations

Resilience will become the competitive advantage