The NCSC research problem book
The most significant problems which the NCSC believes are in need of concerted and significant collaborative effort from multiple and diverse disciplines over the next decade.
Page 28 of 29
AI7 - How can data-driven methods improve cyber security analysis, posture and resilience at national scale?
Bangon Pitipong via Getty Images
Cyber security decisions increasingly depend on making sense of large, varied and imperfect data. At national scale, this can involve information about vulnerabilities, organisations, sectors, technologies, services, incidents and wider dependencies. Manual analysis alone is unlikely to keep pace with the volume, complexity and rate of change.
Data-driven methods, including machine learning, graph analysis, knowledge representation and retrieval-augmented generation could help analysts connect information across sources, identify patterns, reason about risk and prioritise action. However, these methods are only useful if the underlying data is reliable enough, the outputs are explainable, and the uncertainty is made clear to users.
This problem asks how data-driven methods can improve cyber security analysis, posture and resilience at national scale, while remaining robust, interpretable and practically useful.
Strands or sub-problems
Graph-based cyber security analysis and exploration
Cyber security information is often highly connected: organisations use technologies, technologies have vulnerabilities, vulnerabilities affect services, and services may depend on other services. Knowledge graphs and graph data science could help represent these relationships and support analysis such as centrality, link prediction, community detection and risk propagation. Combined with retrieval-augmented generation, these methods could help users explore complex cyber information while preserving links to evidence and provenance.
Handling poor data quality and uncertainty
Data-driven cyber security analysis is limited by the quality of the data it relies on. Data may be incomplete, inconsistent, outdated, duplicated or difficult to match across sources. Better approaches are needed for entity matching, provenance, confidence scoring and communicating uncertainty, so that users understand how much trust to place in an output and where further validation is needed.
Building confidence from weak cyber indicators
Many useful cyber indicators are noisy or individually inconclusive. Weak supervision could help combine signals from multiple sources into more useful indicators, for example when classifying suspicious infrastructure, identifying emerging risks or prioritising further investigation. The challenge is to make these methods reliable enough to support decisions without overstating what weak signals can prove.
Measuring cyber posture and prioritisation
Organisations and sectors need to understand whether their security posture is improving, where the most important gaps are, and which interventions are likely to have the greatest effect. Data-driven methods could help combine information about vulnerabilities, exposure, mitigations, services and dependencies into posture metrics and prioritisation models. These approaches need to be transparent, explainable and sensitive to the context in which decisions are made.
Scalability and reuse
National-scale cyber analysis cannot depend on bespoke, one-off analysis for every question. Methods need to be scalable, reusable and adaptable across different datasets, sectors and use cases. This includes reusable analytical tooling, clear data models, repeatable pipelines and methods that can be maintained as the cyber landscape changes.
Why this is important
As the pace, scale and harm caused by cyber threats continue to grow, it is increasingly important that we are able to harness cutting-edge data science in support of our cyber security mission objectives. The imperative is to understand and illuminate where the UK faces national security and economic risk, and to generate the insights needed for rapid and robust decision-making. However, we know that the data landscape is fragmented and imperfect. This challenge is about how we can still derive actionable insights despite those limitations.
