The NCSC research problem book
The most significant problems which the NCSC believes are in need of concerted and significant collaborative effort from multiple and diverse disciplines over the next decade.
Page 21 of 29
AI in cyber security problems
nuddss via Getty Images
AI is already transforming the cyber security landscape. Defenders have been using AI systems to analyse large volumes of data, triage alerts and to support decision-making. But now, more capable ‘agentic AI’ systems are also emerging that can plan and act with minimal human intervention. These systems can support more complex tasks such as automated system hardening, continuous vulnerability discovery, detection engineering and incident response. Used well, they may help defenders operate at a speed and scale that would be difficult to achieve solely with human teams.
However, we have evidence that capable AI tools are being used by threat actors. AI lowers the barrier to entry for some offensive tasks and increases the speed and scale of others. Cyber activities that previously required specialist skills – writing exploit code, analysing systems or navigating complex technical environments - are increasingly AI-assisted.
This creates a dual challenge: how can defenders make effective use of AI while also managing the security, safety and operational risks it introduces? The NCSC blog 'One does not simply defend agentically' points out that, unlike attackers, defenders must use AI within organisational, operational and governance constraints, making the adoption of autonomous capabilities particularly challenging. Understanding these risks requires comparison with human-led approaches, including where AI may reduce risk, introduce new risks, or alter how the risk manifests itself in practice. Finding answers will be critical for the cyber security community at large, but is also important for the NCSC and the Department for Digital, Culture, Media and Sport (DCMS) work on Cyber Shield, a national-scale, collaborative approach to agentic cyber defence.
To help guide thinking on this, the chapter is organised around three main areas:
-
AI for autonomous cyber defence, including AI systems that help defenders find vulnerability, harden systems, as well as detect and respond to cyber threats
-
Security of AI systems, including how AI and machine learning systems can be assessed, protected and deployed securely and in a resilient manner
-
AI for cyber resilience at national scale, including the use of machine learning, graph analysis, knowledge representation and other data-driven methods to gain insights into cyber resilience at a national scale.
Human-AI collaboration (or human-machine teaming) is a cross-cutting underlying theme throughout this chapter. Research is needed to understand which cyber defence tasks are suitable for different levels of autonomy, where human judgement remains essential, and how to maintain meaningful oversight as systems become more capable and are deployed at greater scale.
The problems in this chapter reflect some of the challenges created by the growing role of AI in cyber security. This list is not exhaustive and instead aims to provide a guide to the types of research questions where academia, industry and government can make tangible progress over the next 5 years.
We have identified the following high-level research questions where significant technical, operational and assurance challenges remain:
- 1
How can organisations re-design cyber defence workflows to maximise the benefits of agentic AI?
- 2
How can AI agents reliably help defenders identify, prioritise and reduce their attack surface?
- 3
How can AI strengthen threat detection, triage and incident response?
- 4
How should autonomous or semi-autonomous cyber defence systems be evaluated and assured?
- 5
How can individual AI and machine learning systems be secured throughout their lifecycle?
- 6
How do we secure current and future AI ecosystems?
- 7
How can data-driven methods improve cyber security analysis, posture and resilience at national scale?
We’re interested to hear about work that’s happening that contributes to solving these AI in cyber security problems and have set up a form for you to tell us more, including the institution you’re linked with and whether you’ve worked with us before.
