Machine learning principles

The use of CL can be a crucial part of helping keep a system secure as well as performant. It allows you to dynamically tackle issues like model drift and erroneous predictions, while also creating a 'moving target' for attackers. CL can come in many forms and is not always obvious. From a security point of view, you can consider CL to be any time that your model's behaviour is affected by data collected during operation.

Although CL can bring important security benefits, the process of taking user input and feedback to retrain a model during operation also opens a new attack vector, as you are allowing the logic and behaviour of your model to be changed by possibly untrusted sources. Retraining on an updated dataset may cause the model's decision surface to change, causing old (potentially correct) behaviours to be lost, or ‘forgotten’, in certain circumstances. Once a model has been retrained, it is not the same as the old model and may well give quite different results, particularly on the edges of the old model's competence. It should therefore be treated as a new model.

The principles in the development part of the life cycle highlight some key points that need to be considered when creating or developing a new model, including when gathering data. Testing processes should be in place to prevent external interactions having a negative effect on your model's behaviour, and updates may need to be rigorously tested in the same way as the original model release. 

CL complicates the tracking of dataset and model updates as outlined in principle 2.3 and makes supply chains more difficult to secure. You should be aware of these risks and have systems or processes in place to mitigate them. Updates to your model and dataset should be validated, tracked and documented in the same way as they are during your development process. This allows you to monitor for anomalous updates from CL manipulation. You should also be able to react to attacks, for example, by having the ability to roll back in the case of collecting poisoned or mislabelled data.

Develop effective MLOps so performance targets are achieved before an updated model goes into production

Creating up-to-date CL models should be an automated process that is driven by MLOps best practice and includes suitable security monitoring. This should allow the tracking of data throughout the CL process including the ability to trace the source of an error or attack (such as poisoning). It's also important to be able to identify and manage drift, where a model may start to adjust its predictions based on new training data that may move outside of the original operating criteria, or a model may begin to degrade as the data changes and exhibits new features or concepts.

There are lots of sources that describe how to conduct effective MLOps and several tools that can be used to create an effective MLOps environment. These include Continuous Machine Learning (CML), Data Version Control (DVC) and MLflow.

Consider using the tracked model metadata and parameters as an alert system. Significant divergence from the original or previous update of an asset may indicate anomalous or malicious behaviour. When a metric diverges by a given amount, a process can be triggered to request human verification. Your application and risk appetite will determine how often your CL updates require manual testing. This may be at each release of an updated/retrained model.

Consider using a pipeline architecture with checkpoints for testing

As a part of this approach, you may consider running multiple models in parallel, enabling models to be updated and tested before being released for operation. A solution for this may be to use sandboxed environments to run updated models for comparison, or if models require user interaction, experimentation deployments can be set up that target smaller groups of users that interact with an updated model.

Capture updates to datasets and models in their associated metadata

During CL you're collecting new data and this should be treated as such. The same security concerns that are present during the gathering of data in the development stage are also present here. It's especially important to think about how much you can trust your data source. Are they trusted users of a system, or can anyone provide new data points? Follow guidance for creating metadata for an asset discussed in principle 2.3.

Consider processing data and training locally

If you don't intend to update a central/global model, CL can be carried out locally on a user's device without the updated model ever leaving the device. When you intend a local instance to contribute to updating a central model, your solution may include techniques such as federated learning. This blog by the government’s Responsible Technology Adoption Unit highlights some considerations related to privacy, security and federated learning.