The NCSC research problem book
The most significant problems which the NCSC believes are in need of concerted and significant collaborative effort from multiple and diverse disciplines over the next decade.
Page 27 of 29
AI6 - How do we secure current and future AI ecosystems?
erhui1979 via Getty Images
AI systems are increasingly part of wider ecosystems rather than isolated tools. They may depend on foundation models, third-party components, datasets, plugins, APIs, agent frameworks, cloud services, deployment platforms and human operators. As these ecosystems become more connected, security risks can arise not only from individual models, but from the way models, tools, people and organisations interact.
This problem looks beyond securing a single AI system. It asks how we can secure the wider AI ecosystem as current systems evolve and new paradigms emerge, including agentic AI, multi-agent systems and AI-enabled supply chains. The challenge is to understand how vulnerabilities, misuse, compromise or unsafe behaviour could propagate across interconnected AI systems, and how organisations can build resilience against those risks.
Securing AI ecosystems will require technical controls, but also governance, monitoring, vulnerability management and socio-technical understanding. The aim is to ensure that AI systems can be adopted safely at scale, without creating systemic risks that are difficult to detect, contain or recover from.
Strands or sub-problems
Emerging AI paradigms
AI systems are continuously evolving. Agentic systems, multi-agent architectures, tool-using models and systems with longer-term memory may create new security properties that are not well covered by existing approaches. It is important to understand how these changes affect threat models, assurance methods and the kinds of controls needed to keep AI systems secure as capabilities evolve.
AI supply chains
AI systems often rely on components from multiple organisations, including models, datasets, code libraries, APIs, plugins, infrastructure and fine-tuning pipelines. Weaknesses or compromise in one part of this supply chain can affect systems built on top of it. Better ways are needed to understand provenance, dependencies and trust across AI supply chains, so that developers and deployers can make informed decisions about what they use.
Vulnerability management for AI systems
Traditional vulnerability management is not always well suited to AI systems. Some issues may relate to model behaviour, training data, fine-tuning, prompts, tool access or deployment context rather than a conventional software flaw. Organisations need ways to identify, report, prioritise and remediate AI vulnerabilities, including deciding when a model, component or integration should be patched, restricted, replaced or withdrawn.
Agentic and multi-agent ecosystems
When AI agents can communicate, delegate tasks or act through tools, risks may emerge from their interactions rather than from any single agent. Multi-agent systems could amplify errors, create coordination failures, or make malicious and unintended behaviour harder to detect. Securing these systems means understanding communication, permissions, shared memory, monitoring and containment across networks of interacting agents.
Human-AI collaboration and oversight
AI ecosystems include people as well as technology. Users, operators, developers, assurance teams and decision-makers all influence how AI systems are used and secured. Human-AI collaboration needs to be designed so that people can understand system behaviour, intervene when needed and avoid both over-reliance and unnecessary rejection of useful tools.
Systemic and socio-technical risk
As AI becomes embedded across organisations and sectors, risks may become systemic. A failure, compromise or misleading output in one part of the ecosystem could affect many downstream users or connected services. Securing AI ecosystems therefore requires attention to organisational incentives, governance, standards, accountability and resilience, as well as technical mitigations.
Why this is important
Increasingly, deployed AI systems are ecosystems. Their security rests on foundation models, datasets, third-party components, agent frameworks, cloud platforms and human operators, and on the interactions between all of these. Autonomy changes the failure modes: a compromised, misdirected or mistaken agent can act outside its authorised bounds, and that rogue behaviour can propagate through delegation chains, shared memory, connected tools, and organisational boundaries at machine speed. Assurance built for individual systems and human-speed oversight has not kept pace. We still lack dependable ways to establish provenance and trust across AI supply chains, test and monitor multi-agent behaviour, manage AI-specific vulnerabilities, and contain and investigate failures. Progress here would give operators practical, evidence-based controls, whilst giving regulators and standards bodies a technical basis for proportionate requirements. This will allow agentic AI to be adopted at scale in critical services with justified confidence.
