NCSC Annual Review 2023

Over the last year, we have all witnessed the significant increase in interest around artificial intelligence (AI) – particularly following the launch of ChatGPT. This has been accompanied by many, often dystopian, predictions about how AI will impact almost every aspect of our lives in the coming years.

While many people will have encountered some varieties of AI such as large language models (LLMs) like ChatGPT, the field of AI is incredibly broad. As the UK’s national technical authority for cyber security, the NCSC has been focused on understanding the cyber security challenges and opportunities that AI presents for many years.

And as this exciting field of technology develops, we will continue to conduct research into AI to understand its vulnerabilities and keep track of how our adversaries are seeking to exploit AI in an irresponsible and unethical manner for their nefarious ends.

While much of the debate around AI focuses on its broad existential risks, there are many immediate security concerns which the rapid development of AI brings. Alongside industry and international partners, we are working to provide clear guidance to understand and manage these risks. We must also remember that while the risks of AI are significant, at its core, AI is a type of software, so while many of the challenges it creates are new, there are also many lessons that we have learnt from previous generations of cyber security practice that we can use to secure this rapidly developing technology.

AI also presents the cyber security sector with significant opportunities to develop new and innovative ways to defend ourselves against hostile actors. Over the coming years, the NCSC will continue to work collaboratively with industry and academia to maximise the benefits of AI to cyber security. 
 

One of the biggest challenges around the cyber security of AI is one that is common to any technology: ensuring that it is ‘secure by design’ and built on secure foundations. As AI becomes more prevalent across the technology ecosystem – and increasingly incorporated into critical systems – we need to ensure that these systems are being designed and deployed securely, to avoid harm to individuals and systems, for example putting personal safety or data at risk.

We must remember lessons from the early days of the internet. In the 1990s, new technology was rapidly rolled out – the world wide web, web browsers, the first search engines, text messages - with very limited focus on security considerations. And we continue to pay the price, for example with the presence of vulnerabilities in core email and web protocols that were not ‘secure by design’.

As AI technologies are rolled out, there are several significant risks that may make our technology ecosystem more vulnerable.

Firstly, if security is only a secondary concern in the development of AI systems, we risk vulnerabilities being designed into new systems.

Secondly, AI will require the development and innovation of existing technology stacks. This development is likely to exacerbate existing vulnerabilities within these tech stacks and introduce new ones. And just as supply chain security is vital in current technology, it will remain incredibly important as AI is integrated into technology stacks.

Thirdly, it is likely that as AI is incorporated into existing IT functions, it could be integrated into legacy hardware, firmware, software, and applications, which may hold outdated security protocols.

AI security must therefore apply across this integration of technology stacks to be not only ‘secure by design’, but also built on secure foundations and to consider security across the whole life cycle of the technology. It requires organisations seeking to implement AI technology within their systems to consider the system as a whole – including the underlying infrastructure and supply chains – and not just the AI component. This requires security to be made a business priority within the supply chain of emerging technology, rather than simply a technical feature. 
 

Most applications of AI are built using machine learning (ML) techniques. ML enables a system to ‘learn’ for itself about how to derive information from data, with minimal supervision from a human developer. But the use of ML creates its own risks.

Training AI using most ML algorithms requires huge volumes of data, but there is no inherent mechanism for filtering out bad, inaccurate, or toxic data. Therefore biases, inaccuracy and misinformation can be intentionally, or unintentionally, built into AI with poor training or poor data. And even if it is wrong, AI can still appear extremely convincing.

As a result of this vulnerability in ML-trained AI, a new category of attack has been introduced that we need to counter: adversarial attacks.

In simple terms, adversarial attacks are an attempt to trick ML algorithms, to influence the outcome of the AI. There are several methods of adversarial attack, including data poisoning attacks, where the attacker attempts to contaminate the data used in the ML process.

While there is significant focus on the risks of AI, we must also ensure that we take advantage of the significant opportunities that AI brings to cyber defenders.

AI is already being used to detect known types of fraud, through the detection of anomalies in user actions. In consumer banking, this can be applied to improved monitoring of card usage, more quickly blocking fraudsters from using another user’s credit card by identifying strange individual transactions. AI will be able to improve detection and triage of cyber attacks. As AI detects patterns and relationships between data, it can be used to recognise phishing emails and cluster them to identify campaigns, which are then more easily mitigated.

It can be used to support cyber defenders, with analysis of logs and files, network traffic, supporting secure code development and testing, and threat intelligence. LLMs, in particular, are proving to be beneficial in finding vulnerabilities in source code and potentially spotting – and even fixing – flaws before attackers get the chance to exploit them. AI is incredibly quick, so could be used to pick up on potential attacks more rapidly if a vulnerability is exploited, speeding up the process of finding and fixing security vulnerabilities, and making malware analysis more efficient. Over time, it is likely we will see AI providing a generation of more secure code through faster learning.

However, not all of these cyber security improvements will come automatically. We need to foster a community that encompasses the entire cyber security ecosystem and focuses on growing this sector in a way that is diverse and inclusive. We also need to ensure that where AI is used to enhance cyber security that we are doing all we can as a community to avoid introducing and reinforcing bias into cyber security analysis and threat monitoring. That is why the NCSC is working closely with the Alan Turing Institute to both help develop and benefit from research on AI and cyber security across a range of topics. 
 

As we have already highlighted, AI models have new, inherent weaknesses and vulnerabilities which need to be understood by those developing them. Some cutting-edge AI models can be incredibly complex – often even their creators don’t fully understand exactly how they work or what happens inside the model. This lack of ‘explain-ability’ is one of the key safety and security challenges.

Another central challenge is around the security and confidentiality of users’ data. The fundamental operation of AI systems relies on continued access to large, representative and often sensitive datasets – this goes against normal cyber security approaches of restricting access to sensitive systems and components.

Some of the risks that flow from this are straightforward; for example, through either malicious activity or accidents, confidential information could be leaked. But there are other data risks; for example, AI models can allow adversaries to reconstruct the data they were trained on through querying the models. It’s not only the integrity of the output or what it can do that is important; the data and models of the AI are valuable assets in and of themselves and should be appropriately protected.

AI has the potential to dramatically change the scale of the cyber security challenge that we face. Hostile adversaries are already using LLMs to develop increasingly sophisticated phishing emails and scams.

In the coming years, AI could be used to conduct targeted or untargeted cyber attacks and it is also likely to lead to the further proliferation of cyber capability to a wider range of actors. Generative AI also has the potential to create synthetic cyber environments which could be used for criminal purposes or fraud.

As the opportunities of AI become more obvious, an increasing number of organisations are seeking to use it. It is vital that as they develop AI capabilities, they understand the heightened and novel risks they are running by doing so – and how best to mitigate them.

The NCSC has already provided guidance to organisations seeking to integrate LLMs into their business operations. Our understanding of the capabilities, weaknesses and vulnerabilities of LLMs will continue to develop as use cases and applications of the technology increases. As a result organisations should make sure they are comfortable with the ‘worst case scenario’ of whatever the LLM application is permitted to do.

As the UK national technical authority for cyber security, the NCSC’s role is to understand and promote the cyber security of AI technologies, working with government, academia and industry.

The NCSC has published, and will continue to publish, guidance to support a range of different groups – from cyber security professionals to business leaders – as they seek to understand and realise the benefits that AI offers.

A number of alumni companies from the ‘NCSC for Startups’ programme are using AI in a variety of ways. Meterian uses AI to boost its speed and comprehensiveness of indexing open source vulnerabilities to give enterprises the best visibility and auto-remediation of open source supply chain risks when using programming languages as old as Perl, C/C++ or the next generation language Rust. Lexverify uses AI (advanced NLP) for real-time prevention of legal, compliance, and cyber risks on electronic communications, and Visible uses AI-generated reports to provide highly detailed insight into how individuals are perceived online.

The NCSC itself is also seeking to make use of AI as part of our mission to keep the UK the safest place to live and work online. We are currently using machine learning to spot complex patterns of activity across multiple ACD datasets. For example, correlating events from our protective DNS service with those from our host-based logging capability to identify hidden malicious behaviour. We are also investigating new opportunities for ACD to incorporate improved human-AI teaming, as well as researching the potential for autonomous capabilities in the future.

Soon, we plan to use AI to more effectively spot mutated forms of malware to enable the identification and release of indicators of compromise (IOCs) more quickly than traditional software reverse engineering or code matching allows. We also plan to identify patterns in the use of commodity services – like blockchain-based DNS – used by malware actors to flag potential IOCs before they have even gone live. Longer term, we plan to use the huge volumes of data generated by the NCSC’s ACD products and services to identify obscure patterns of malicious behaviour across the entire government technology estate among other areas.