Understanding adversarial attacks against Machine Learning and AI

is a sub-field of Artificial Intelligence (AI). ML algorithms allow computers to recognise patterns in data, modelling the relationship between the data they are given and the problem the algorithm designer is trying to solve, without having to be explicitly programmed by a human. These relationships and the software to operate them, are captured in an ML model which can be used to predict the most likely responses to new data points.

The process of learning those patterns in data relevant to the designer’s problem (or task), is known as training the model.

The process of learning those patterns in data relevant to the designer’s problem (or task), is known as training the model.

The model’s architecture refers to the specific algorithmic design and structural features of the ML model that define how it is trained and how it operates. This includes details of different model components such as the configuration of layers in a neural network or the decision branching structure in a decision tree model, and parameter settings.

are a prominent class of ML models, which use advanced model architectures and are trained on massive datasets to interpret complex inputs (known as prompts) and generate human-like text or code. LLMs are the core technology behind the development of agentic AI systems, which are enabled with wide system and tool accesses to answer a broad set of user tasks.

incorporate all the hardware and software required to operate and serve the ML model(s) for real usage, commonly accessed by users and other systems through an application programming interface (API), chatbot or integrated service.

is the implementation of a trained ML model in a live environment for operational use, with appropriate monitoring and maintenance.

refers to an individual or group performing a deliberate attack on an ML model or ML-enabled system to achieve their goals, such as compromising its confidentiality, integrity or availability, in order to disrupt model functionality, gain unauthorised control, and/or extract private information. An actor may also aim to introduce hidden malicious functionality to a model that allows them to alter its outputs when a given input is provided or a condition is triggered, while laying dormant until triggered.

are intentional attacks against ML systems executed by a malicious actor to exploit vulnerabilities specific to the architecture, training and/or operation of the ML model.

are individual techniques which can be used to attack ML models. These range from simple to complex, may be single tasks or multi-step chains, but they focus on accomplishing a single goal for the malicious actor. The examples of attack techniques provided in section 2 have been drawn from academic papers, industry papers, online discourse or observed attacks.

group AML attack techniques based on how they interact with the ML system and which ML‑specific vulnerabilities they exploit.

Model characterisation refers to techniques used to analyse and interpret the behaviour of a model when the malicious actor does not have full knowledge of the system. The aim may be to:

• Replicate the model’s functionality.
• Reveal confidential or proprietary information about the model's provenance or training.
• Identify vulnerabilities that could be exploited later.
• Create a representative surrogate of the model for attack testing.

Since later attack classes rely heavily on understanding the model, protecting against model characterisation is critical. This is especially important for attacks that may be aided by the use of a surrogate model, where a malicious actor builds a close approximation of the target model and uses that to develop attacks without the need to remain covert.

This class comprises only those attacks that exploit ML-specific weaknesses. Other reconnaissance activities are not included, although they may also benefit a malicious actor. These include scraping publicly available information such as model cards and documentation, and observing behaviour during standard interaction with a model.

This attack class goes beyond measuring performance. It focuses on reconstructing or inferring how and why a model makes specific decisions, which improves a malicious actor’s understanding of the target ML model, or enables them to steal some aspect of its functionality. Different attack techniques may require full access to the internal components of the model, a partial view, or query-only access to the model, and techniques can be applied through:

• Static analysis – examining the model without running it.
• Dynamic analysis – running the model and testing data against it.

A malicious actor may seek privileged details about the model such as:

• parameter weights
• hyperparameters
• provenance of base models used to build surrogate models
• training data and output classes
• fine-tuning relationships
• linked resources, environmental constraints, and wider system design

The following attack techniques are demonstrated by the linked papers:

• Discovering key model information
  • Stealing Part of a Production Language Model
• Determining model provenance
  • Neural Phylogeny: Fine-Tuning Relationship Detection among Neural Networks
• Building a surrogate model from available information to enable later attacks
  • Knockoff Nets: Stealing Functionality of Black-Box Models
• Identifying common hallucinations
  • We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs

Model inversion attacks recover information about the data linked to a model. This could involve:

• Estimating the data it was trained on.
• Retrieving the data it is operating on.
• Learning the relationship between the model’s inputs and outputs.

This can expose confidential capabilities, commercial secrets, or weaknesses in the model or personal data. Malicious actors don’t always need exact data; partial recovery or indicative patterns can still meet their objectives or enable later attacks.

These attacks exploit knowledge of the model and its outputs to infer confidential information. Common techniques include:

• Reconstructing or identifying training data points from queries to the model.
• Inverting embeddings to recover user input data, potentially by training a model to map outputs to inputs.
• Distilling a model based on the malicious actor’s limited view of model inputs and outputs.

Even if complete data recovery fails, inversion can reveal relationships between inputs and outputs, enabling further attacks like model input manipulation.

The following attack techniques are demonstrated by the linked papers:

• Unauthorised distillation/model stealing
  • I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences
• Decision boundary detection
  • Stealing Machine Learning Models via Prediction APIs
  • Understanding the Decision Boundary of Deep Neural Networks: An Empirical Study
• Membership inference of data
  • Membership Inference Attacks From First Principles
• Training data reconstruction
  • Reconstructing Training Data from Trained Neural Networks
• Extraction of data via inference
  • Extracting Training Data from Large Language Models
• Inversion of embedded data
  • Variational Model Inversion Attacks
  • Text Embeddings Reveal (Almost) As Much As Text 

Training data poisoning is a class of attack where the data a model is trained upon is intentionally manipulated or corrupted. This manipulation compromises the model's training, which will alter how well the model performs at its trained task, or introduce new aspects to its behaviour. Simple attacks may lead to poor inferences, delays and wasted training time, while more complex attacks may create intentional inaccuracies, biased outcomes, or malicious functionality within the model.

This class only covers training data on which the model is trained or is fine-tuned. In the case of active learning or reinforcement learning systems, where inputs and outputs are captured and used to refine a model, this class of attack may intersect with model input manipulation.

In many cases, only minimal understanding of the model is needed to craft ‘poisonous’ data points. Legitimate data points (including labels) can be replaced or modified to suit a malicious actor’s intent. Other techniques may involve removing or rearranging specific data points within a training dataset to introduce unwanted biases in the model.

The following attack techniques are demonstrated by the linked papers:

• Publishing poisoned data sets
  • Poisoning Web-Scale Training Datasets is Practical
• Publishing a poisoned base model
  • Model Supply Chain Poisoning: Backdooring Pre-trained Models via Embedding Indistinguishability
• Targeted data poisoning (for example, of a class or decision boundary)
  • Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
  • BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
• Broad data poisoning (for example, general degradation or across a huge dataset)
  • Poisoning Web-Scale Training Datasets is Practical
• Including watermarks to reveal use of specific data
  • Copyright Traps for Large Language Models

Malicious model training attacks manipulate aspects of a model’s training process to alter the output model. Changing the training hyperparameters or reworking the architecture, for example, can alter the model and result in a malicious actor achieving their desired effect.

Some methods of malicious model training include:

• Changing the training epochs.
• Altering the learning algorithms.
• Changing the activation functions.

When a malicious actor uses these methods, some may result in an output model fundamentally different from the intended model, but other implementations may still produce models that run with the same inputs and outputs.

A malicious actor performing these attacks may be able to compromise the model's integrity by degrading its overall performance or cause resource depletion through model retraining. In rare circumstances, it may also be possible to embed unwanted behaviours in the model.

The following attack techniques are demonstrated by the linked papers:

• Malicious fine-tuning
  • Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
• Altering the training regimen of a model
  • Planting Undetectable Backdoors in Machine Learning Models
• Altering model compression
  • Exploiting LLM Quantisation
• Introducing hidden functionality in the architecture during training
  • The Dark Side of AutoML: Towards Architectural Backdoor Search

Model input manipulation techniques involve the generation of input data which usually appears to be benign but has been maliciously modified.

These modifications can range from simple input transformations that change the model inference (commonly known as model evasion attacks), to carefully crafted inputs which exploit vulnerabilities in the model's inference process, causing it to produce incorrect or harmful outputs outside of its intended functionality. Some attacks in this class may also be able to achieve an unwanted effect on the model, such as allowing an unwanted output to be produced or forcing the model into a repetitive execution loop.

Against LLMs, model input manipulation is most commonly implemented as prompt injection techniques, where malicious actors attempt to breach LLM usage controls by concatenating their input prompts with instructions designed to minimise the chance of an LLM refusing to respond. As these systems are commonly supported by reference content databases, they are also at risk of indirect prompt injection attacks, where malicious content may be supplied in the model’s prompt-enriching queries to those databases.

Similarly, models are increasingly linked together and able to call each other’s APIs, as in the development of agentic AI systems, so model input manipulation attacks could also originate from system inputs, tool responses or other agentic systems.

Some techniques in this class may alter a malicious input to activate some previously hidden functionality in the model – see training data poisoning and model artefact manipulation – which may demonstrate the linked attack chains of a sophisticated actor.

The following attack techniques are demonstrated by the linked papers:

• Crafting adversarial perturbations
  • Towards Evaluating the Robustness of Neural Networks
• Direct prompt injection
  • Ignore Previous Prompt: Attack Techniques For Language Models
• Indirect prompt injection
  • Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Splitting prompt injection across multiple modalities
  • Manipulating Multimodal Agents via Cross-Modal Prompt Injection

Model artefact manipulation attacks involve altering a trained model after training to change its behaviour. The simplest form of attack is to delete or corrupt the model, but most attacks aim to subtly modify the model so that it continues to function but behaves in a way intended by the malicious actor.

The most prominent technique in this attack class is changing the parameters of the model, inherently altering its functionality. Other examples include modifying the model’s architecture, layers or activation functions. Using these techniques, a malicious actor may seek to:

• Affect model performance globally.
• Introduce a specific bias.
• Extend an existing performance gap.

Other techniques in this class may:

• Edit the computation graph of the model to introduce hidden or more complex functionality, masking the introduction of malicious functionality.
• Apply a mask to a model’s weights to hide capability in the least significant bits of the model.
• Modify resources associated with the model, for example a tokeniser file, which may be less-well protected than the model weights.

Model artefact manipulation attacks may be carried out at any point in the deployment of a model, but may represent a significant supply chain risk when the origin of the model is not fully known.

The following attack techniques are demonstrated by the linked papers:

• Altering model weights and biases
  • Refusal in Language Models Is Mediated by a Single Direction
  • EvilModel: Hiding Malware Inside of Neural Network Models
• Altering the model’s architecture
  • Architectural Backdoors in Neural Networks
  • Architectural Backdoors in Deep Learning: A Survey of Vulnerabilities, Detection, and Defense
• Altering source code libraries to affect the running of the model
  • ImpNet: Imperceptible and blackbox-undetectable backdoors in compiled neural networks
• Altering non-weight model components
  • When Tokenizers Drift: Hidden Costs and Security Risks in LLM Deployments

Model hardware attacks exploit vulnerabilities in the physical infrastructure and hardware components underpinning the training, storage or execution of models, to better understand or indirectly affect those models.

This attack class targets the hardware hosting the model, providing an alternative attack surface to the software-based attacks outlined above. In cases where the hardware is virtualised away from the ML system, such as in a cloud environment or Kubernetes deployment, a malicious actor may be able to replicate some hardware attacks through manipulation of the virtualisation code which controls hardware allocation.

A malicious actor may be able to achieve multiple goals with hardware attacks, many of which affect the entire system rather than just the model. This class focuses on attacks that either reveal unique insights into the operation of the model – for example, GPU side-channel attacks leading to information leakage – or potentially disrupt inference performance.

The following attack techniques are demonstrated by the linked papers:

• Side-channel attacks to understand the model
  • Open DNN Box by Power Side-Channel Attack
• Physical fault injection to hardware altering model logic
  • Practical Fault Attack on Deep Neural Networks
• Causing changes in the hardware’s RAM to affect model predictions
  • GPUHammer: Rowhammer Attacks on GPU Memories are Practical