Machine learning principles
Page 4 of 22
1.2 Model the threats to your system
Goals:
-
You understand how your ML model should perform, and design processes to identify and/or correct if it fails (either unintentionally or because of an adversary).
-
You understand the implications of attacks on your ML model and the effects on wider system behaviour.
-
You have sufficient expertise for your whole system design and application, not just AI/ML knowledge.
-
You understand the wider consequences if your system is compromised (including reputational damage for your organisation).
Why is this important?
An ML model is often just one of many components in a system, and how it interacts with the wider system should be considered when planning any ML deployment. It is important to understand how an attack on the confidentiality, integrity and/or availability of an individual ML component impacts the wider system, and to use this understanding to inform design decisions up and down stream from the model. The impact on other assets, systems or processes (such as dataset confidentiality) should be considered here too.
It will never be possible to guarantee complete security against attackers, whose are always developing new techniques. New attacks on ML systems are demonstrated regularly, and there are significant limitations to existing defence methods. So it’s important to ensure that you understand the potential impact on your wider system, should an ML component fail. Scenarios to consider should include:
-
how your ML system is used in normal operation, and what would happen if its error rate spiked (following an attack, the introduction of out-of-sample observations, or data drift)
-
how you would identify such an error rate (you can't rely on the ML model to do this itself)
-
how your ML component fits into your larger system (and what would happen if the component no longer operated as intended)
Exploring these scenarios will help system designers decide whether additional mitigations are required to prevent undesirable behaviour if the ML model is attacked.
How could this principle be implemented?
Create a high-level threat model for the ML system
You can create a high-level threat model to gain an initial understanding of the wider system implications of any attack on your ML component. Assess the implications of ML security threats and model failure modes across the entire system. A baseline threat model such as that suggested by OWASP can then be refined during development.
Use the CIA (confidentiality, integrity, availability) triad and the high-level threat model to explore the system. This may be ‘baked in’ natively to the wider system design (including ‘humans-in-the-loop’), or may require you to make design decisions based on ML component's limitations and your risk appetite.
An effective way of capturing and sharing the necessary ML model information may be to treat it as a component in the wider system. Documenting inputs and outputs and highlighting limitations (such as uncertainties and value bounds) can be an effective way to do this.
Model wider system behaviours
It's important that both ML expertise and wider system/domain expertise are used in system design, and that people understand the limits of your ML components.
A common design pattern is to implement layers of countermeasures that work in concert to determine genuine requests or identify dubious activity. This could include logic / rule-based controls outside of the model itself, for example:
-
expert systems: use explicitly defined 'if-then' rules
-
subjective logic: a type of probabilistic logic that factors uncertainty into decisions
-
decision trees: a branching structure where each node represents a test, the outcome of which dictates the next test
Finally, consider user access and design your system accordingly. A web-hosted system whose source code is publicly available may need more defensive measures than a closed, proprietary system that can only be accessed through a controlled interface.
The NCSC has guidance on understanding system-driven risk management. Also, NIST provides risk management guidance with a focus on AI.
