Machine learning principles
Page 3 of 22
1.1 Raise awareness of ML threats and risks
Goals:
-
Security is considered a key part of your ML project and integrated into your workflows at all stages of the system’s life cycle.
-
ML practitioners, data scientists and software developers are familiar with the inherent vulnerabilities and failure modes in ML workflows and algorithms, and follow best practices to mitigate known risks.
Why is this important?
‘Security by design’ is a key principle in traditional software development, and requires significant resources throughout a system’s life cycle. However, adopting this approach from the start will prevent costly redesigns later. It means developers must invest in prioritising features, mechanisms, and implementation of tools that protect customers at each layer of the system design, and across all stages of the development life cycle.
In addition to ML-specific threats, your teams should have experience across the range of disciplines or domains required for secure system design more generally. This is particularly important when highlighting the limitations of ML components to wider system designers.
When developing systems with an ML component, it's important that threats and mitigations specific to ML systems are understood alongside non-AI software security standards. Studies - such as this one from Cornell University have shown that many ML practitioners aren't aware of the specific vulnerabilities of many ML algorithms, or don't have the right tools to assess these vulnerabilities in the first place.
In addition, as the push to low code and automated ML continues, it's also important to think about the background and skills of people using these systems, who are less likely to be 'developers'. Although they may not consider themselves to be writing programs or systems, they are still using underlying ML techniques that contain security vulnerabilities. Understanding these will help them make the right design decisions, that are proportionate to a project's security requirements.
How could this principle be implemented?
Provide guidance on the unique security risks facing AI systems
Research on ML security is fast-moving and you should make sure that practitioners receive the right support to keep their knowledge up to date, including training on the threats that are unique to ML components. For example, developers should be aware of the different types of threats their ML systems may be vulnerable to, which include:
-
Evasion attacks:
techniques to evade the correct behaviour of an ML system by providing a specific input designed to cause a failure. Examples include adversarial examples and prompt injection.
-
Poisoning attacks:
introducing malicious data into a model's training process to make the model malfunction given a specific input, degrade performance more generally, or introduce backdoors that can be exploited later.
-
Privacy attacks:
obtaining confidential information by repeated or targeted querying of a model. A model extraction attack attempts to estimate model parameters or build a copy of the model. A data extraction attack attempts to determine the data a model was trained on (membership inference) or extract samples of training data from a deployed model (model inversion).
Technical team members working on model development should be familiar with a range of attacks on ML systems. Some of these attacks are highlighted in:
- Microsoft’s blog on Failure Modes in Machine Learning
- The National Institute of Standards and Technology (NIST) Adversarial ML: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2 E202 3)
- Germany's Federal Office for Information Security (BSI) AI Security Concerns in a Nutshell report
- The MITRE ATLAS framework (ATLAS is a knowledge base of adversary tactics, techniques, mitigations and case studies for ML systems based on real-world observation, demonstrations from ML red teams and security groups, and the state of the possible from academic research)
Enable and encourage a security-centred culture
Developers aren't necessarily security or usability experts, but understanding where decisions affect security, and how to design and implement a solution that works for its intended users, are crucial parts of ensuring that security works in practice. It's therefore important to establish a positive security culture, supported by leaders, providing sufficient credibility so that security is considered right from the start of a project.
In addition, security is not always part of formal data science or ML course curricula, and expertise in these areas may not translate to knowledge of system security. Encourage a cross-discipline culture that recognises the need to work collaboratively with security specialists and subject matter experts, and put in place processes and procedures to share knowledge and experience where disciplines meet and overlap.
A security-centric culture recognises the importance of training and requires all stakeholders involved in the ML life cycle, from requirements to operational use, to understand the threats to a system, including ones unique and inherent to ML. You should therefore be prepared to invest time and resources to promote this. The NCSC has guidance on growing a positive security culture and secure development and deployment. In addition, the European Union Agency for Cybersecurity (ENISA) has published a framework for good cybersecurity practices for AI.
Encourage best security practices by making sure security reviews are integrated into your system life cycle processes. Each application's development life cycle will be different and it's important to integrate security reviews appropriately.
For example:
-
if you take an agile approach to development, consider integrating security reviews into each sprint
-
if you use a machine learning operations (MLOps) approach to automate workflows, ensure that this includes the security of the products you are developing
If teams are aware of and accountable for security-related responsibilities through the life cycle, it encourages a positive security culture. Depending on your specific development process, automated security testing may be an option, although it's still important to have a security-centred culture.
