Preserving integrity in the age of generative AI

Being able to trust what we consume online is a problem that’s been exacerbated by the growth of artificial intelligence (AI).

Whilst AI systems have the potential to bring many benefits to society, the widespread availability of AI and machine learning tools – including generative models and deepfake technologies – means anyone can create or modify data (so text, images, voice or video) with minimal effort, low cost, and increased realism.

Put simply, it is becoming harder to distinguish between what is fake online content, and what isn’t.

Content Credentials is an emerging technology that seeks to counter this erosion of trust. The technology aims to establish the lineage of data, including its source and editing history over time, essentially enabling authenticity to be preserved and verified.

Our US Colleagues in the National Security Agency (NSA) are today publishing introductory guidance on Content Credentials, which we and other international cyber security partners endorse. This is an important, but embryonic, topic.  

The proliferation of generative AI tools is already being used to impersonate, clone and deceive people and systems. The implications could be wide ranging, such as amplifying the lack of trust in data and media, reputational damage, or helping criminals to create more convincing spear-phishing cyber attacks. In December 2024, the FBI warned of the use of AI by cyber criminals using generative AI to create documents and images to share with victims in private communications that they were speaking to a real person, rather than a criminal actor.

The NSA’s guidance highlights a range of other challenges for AI systems. This includes ‘model collapse’, where systems are trained using data generated by a previous version of the model. Instead of improving content, the AI starts to make mistakes that compound over generations, much like a repeatedly photocopied document that degrades over time. This leads to outputs that are increasingly distorted and unreliable.

AI tools designed to detect synthetic or inauthentic data can be ineffective and unreliable, meaning other technical measures are required to provide layered defences for organisations, individuals and society. Content provenance techniques could have an important role to play in strengthening the integrity of information across a range of systems. Content Credentials can help organisations, systems and users make informed decisions about the data they consume, but by themselves, will not solve the problem entirely. However, as the technology evolves, it will help systems to better evaluate the authenticity of content.

As the NSA guidance points out, what is required is a re-examination of the integrity of the entire online information ecosystem, so that provenance of data can be assured, be that for text, image, voice or video data. Effective development of watermarking and provenance standards could raise the bar for criminals and state actors seeking to exploit inauthentic data or media in their cyber attacks.

As is typical with new and maturing standards, it can take time to implement them across various modalities and address new concerns and edge cases as they arise. However, that should not deter organisations from preparing and getting started now, especially for common use cases. The NCSC will be exploring this topic in more detail, as improving the integrity of online information is a major part of making the UK a safer place to live and work online.

Ian McCormack

Deputy CTO