The NCSC research problem book
The most significant problems which the NCSC believes are in need of concerted and significant collaborative effort from multiple and diverse disciplines over the next decade.
Page 25 of 29
AI4 - How should autonomous or semi-autonomous cyber defence systems be evaluated and assured?
Sorbetto via Getty Images
Autonomous and semi-autonomous cyber defence systems may be able to support complex defensive tasks, from triage and investigation through to recommending or taking action. As these systems become more capable, organisations will need evidence that they are effective, reliable and safe enough to use in real cyber security settings.
Evaluating these systems is difficult. A system that performs well in a benchmark or demonstration may behave differently in a live environment, when faced with incomplete data, ambiguous evidence, adversarial behaviour or unusual operational constraints. Assurance also needs to cover more than technical performance. It should consider logging, monitoring, explainability, human oversight, misuse, loss of control and misalignment. All of these are critical to enabling users to have trust and confidence in the AI system.
This problem asks how autonomous and semi-autonomous cyber defence systems should be tested, monitored and assured before and during deployment.
Strands or sub-problems
Evaluation environments and baselines
Cyber defence systems need to be tested in environments that reflect the complexity of real operations without creating unnecessary risk. Simulation, emulation, cyber ranges and controlled real-world settings may all have a role. Research is needed to understand which environments are most useful for different defensive tasks, and how autonomous systems should be compared with human-only teams and human-AI teams when making deployment decisions.
Robustness, reliability and safety
Autonomous and semi-autonomous systems may face noisy data, incomplete evidence, changing environments and deliberate attempts to mislead them. Research is needed to understand how robust and reliable these systems are under realistic conditions, including how they fail, when they should defer to a human, and what safeguards are needed to prevent unsafe, disproportionate or uncontrolled actions.
Monitoring, logging and misuse controls
Assurance should continue after deployment. Autonomous systems need appropriate logging, monitoring and review so that organisations can understand what the system did, why it acted, and whether its behaviour is changing over time. This should also include consideration of how these systems could be misused, including through external compromise, accidental misuse or malicious insiders, and what controls or audit mechanisms are needed to reduce those risks. Monitoring should also help identify when operators are losing effective control over a system’s behaviour or when the system is acting outside authorised bounds.
Explainability and operational confidence
Defenders need to understand enough about a system’s actions and recommendations to trust it appropriately. This does not mean every internal step must be fully transparent, but explanations should help users assess evidence, uncertainty and likely consequences. Research is needed into what forms of explanation support operational confidence without overwhelming users or creating false assurance.
