The NCSC research problem book
The most significant problems which the NCSC believes are in need of concerted and significant collaborative effort from multiple and diverse disciplines over the next decade.
Page 26 of 29
AI5 - How can individual AI and machine learning systems be secured throughout their lifecycle?
useng via Getty Images
AI and machine learning systems should be understood as part of the wider systems into which they are integrated, not as standalone models. Once deployed, they become part of the system’s attack surface, with dependencies on data, training processes, model weights, software, deployment environments, monitoring arrangements and the people responsible for operating them. They also introduce AI-specific risks, such as poisoned data, unsafe fine-tuning, tampered model weights, insecure deployment and unexpected behaviour once in use.
Securing these systems in practice is difficult because AI security controls often involve trade-offs. A mitigation that improves robustness may affect performance, cost, explainability, fairness, usability or operational speed. Organisations therefore need practical ways to decide which controls are appropriate for a given system, threat model and deployment context.
This problem asks how AI and machine learning systems can be secured throughout their lifecycle, while accounting for the constraints and trade-offs that arise in real-world deployment.
Strands or sub-problems
Securing the AI lifecycle
An AI system is shaped by much more than the model that is eventually deployed. Its security depends on the data, training and fine-tuning processes, model weights, configuration, software dependencies, and deployment pipelines that support it. Organisations need a clearer view of how risks arise across this lifecycle so they can apply proportionate controls throughout development and operation, rather than treating deployment as the only point of concern.
Monitoring deployed models
Once an AI system is deployed, how do we know it is still behaving as expected? A model may be affected by drift, misuse, adversarial inputs or changes in the environment around it. It may also fail in ways that are difficult to spot from ordinary system monitoring. Organisations need better ways to decide what to log, what to monitor and what changes should trigger investigation. This would help them detect security-relevant changes and respond when trust in a model is reduced.
Understanding security trade-offs
Improving one security property of an AI system may make another property worse. A control that improves robustness or privacy might reduce task performance, increase cost, make the system harder to explain, or affect fairness and usability. For practical deployment, organisations need to understand these trade-offs before choosing mitigations. Better evidence in this area would help developers and deployers decide which controls are appropriate for a specific system, threat model and use case.
Securing models in constrained environments
Many AI systems will not be deployed in ideal conditions. They may run on edge devices, embedded systems or platforms with limited compute, memory, power or connectivity. They may also be compressed or adapted using techniques such as quantisation, pruning or distillation. These choices can affect both performance and security. We need to understand how deployment constraints change the threat model, and how models can be protected from tampering, reverse engineering or other attacks in constrained environments.
Practical assurance and reuse
A bespoke security assessment for every AI deployment is unlikely to scale. Organisations need assurance methods that can be reused, adapted and understood across different AI systems and deployment contexts. This could include common evidence requirements, reusable tooling, and clearer ways to demonstrate that an AI system is secure enough for its intended purpose. The aim is to make AI security assurance practical for real organisations, not just possible in principle.
Why this is important
Securing AI and machine learning systems is not a point-in-time activity, but an ongoing challenge spanning system design, model development, deployment, operation and retirement. Vulnerabilities can occur at every stage of this cycle, whether introduced through training data, insecure development practices, weaknesses in software supply chains, or failures in monitoring and governance once systems are deployed. AI systems will be increasingly embedded into critical infrastructure and relied on to support complex decisions. To protect all of us, it is vital to ensure that these systems will be secure by design.
Research is needed to develop approaches for securing models and the systems containing them in practical, usable ways, suited to the purposes for which they will be used, and to find ways to maintain assurance of them once they are deployed into complex real-world environments. Progress in these areas will enable organisations to adopt AI with greater confidence, ensuring that innovation is accompanied by the security, accountability and resilience required for its safe and effective use at scale.
