Strengthening national cyber resilience through observability and threat hunting
How organisations can improve their ability to both detect and discover cyber threats.
Whether organisations manage their own digital estate and assets or work with external partners, the NCSC has found significant variation in organisations' ability to:
- monitor their systems (their ‘observability’) and
- proactively hunt for threats
In the vital pursuit of raising the national ability to detect and discover cyber threat, this needs addressing.
In this blog, we urge organisations and their external providers to develop and/or optimise both their observability and threat hunting capabilities, and set out how they can achieve this.
Definitions
Observability means having a clear and detailed view of everything happening across your networks, systems and services.
Threat hunting is the proactive process of searching for signs of cyber threats and intrusion outside of those detected by existing rules. It involves forming educated theories – or hypotheses – based on attacker behaviour and objectives, threat intelligence, or unusual patterns, and then using available data and insights to prove or disprove them.
You can’t hunt what you can’t see
Observability and threat hunting are core and interdependent components of modern cyber defence. Maturing capability across both of these components is essential to strengthening our national cyber resilience.
Having comprehensive observability across an organisation’s networks, systems and services provides the essential foundational data needed to facilitate effective threat detection and hunting. Without this visibility, some areas of the system remain hidden – ‘dark corners’ – making it harder to spot unusual or malicious activity, or investigate suspected breaches.
These dark corners can include user account activity, devices, networks, applications, and cloud services. They may also involve unapproved or unknown technologies or systems, or unofficially-operating 'shadow IT'.
To be optimal, threat hunting relies on extensive observability. Without comprehensive data and visibility, there is a reduced opportunity to detect threats, making it harder to validate hypotheses or uncover hidden intrusions. In short, you can’t hunt what you can’t see.
Aim for comprehensive observability
In many cases, organisations' access to data about their digital estate is patchy or incomplete. For example, an organisation might have tools that monitor computers and networks, but lack visibility into user identities along with their activity or cloud services. Gaps like these make it harder – or impossible – to detect certain suspicious activity.
In addition, even when organisations do collect data, they often can’t search (or hunt) across it, or analyse it all together with complex analytics, which is what adds most value.
The NCSC recommends the following ways to improve how organisations monitor and understand what’s happening across their digital systems:
- Maximise your visibility of systems as well as your ability to query across combined data sets. These data sets should span networks, hosts, devices and services which are on premises as well as cloud. Even if it’s not possible to achieve complete visibility in legacy or niche systems, ensuring some is an imperative, for example having visibility of the network if not endpoints.
- Encourage your technology vendors to follow the NCSC’s guidance on how to build systems that support better monitoring and investigation. This helps shine a light on the dark corners of networks.
In addition, for those producing and updating protocol standards:
- Consult RFC9424 to support appropriate enterprise network level observability.
Mature your threat hunting capabilities
1. Don’t limit yourself to Indicators of Compromise
Organisations often use Indicators of Compromise (IOCs) such as IP addresses, domain names, and file hashes because they’re easy to use and widely supported by security tools.
However, while IOCs are helpful for detecting known threats, they can limit an organisation’s ability to effectively detect and respond to contemporary cyber threats. Attackers can quickly change or hide these indicators using techniques like Fast Flux, cloud-based infrastructure, or ‘living off the land’ (using legitimate tools for malicious purposes) as demonstrated by the scale of LOLBAS. This makes IOCs fragile and short-lived.
The Pyramid of Pain (2013) shows that relying only on IOCs makes it easier for attackers to avoid detection.
2. Develop use of Tactics, Techniques and Procedures
To ensure resilient operations, organisations need to go beyond IOCs and develop effective use of Tactics, Techniques and Procedures (TTPs) which reveal how attackers operate, not just what they use. TTPs provide deeper insights into attacker behaviour, are less fragile, and support proactive threat hunting and long term cyber defence strategies.
Examples of TTPs include:
| Technical pattern | Plain English |
|---|---|
| SSH connections to a high TCP port to a [geographic] network from core network equipment with inverted volumes of expected traffic | A secure connection (SSH) is being made to an unusual port number on a network in a specific country, and it's coming from important internal devices (like routers or servers). The amount of data going in and out is the opposite of what you'd normally expect. Maybe more data is being sent out than received, which could be suspicious |
| [this] command with [this] parameter and parent process run on [operating system|platform] | A specific command was run with a certain option or setting, and it was started by another program (its “parent”) on a particular operating system |
| [this] tool communicating with [this service] accessing files of [this type] | A program or script is talking to a service (like a cloud storage or database) and working with a specific kind of file, for example, downloading .pdf files or uploading .csv spreadsheets |
| access to [hosts|devices] of this [type] via [interface] using [CVE-XXXX-XX] writing files to [path] | Someone is connecting to certain types of devices (like printers, servers, or IoT gadgets) through a specific method (like a web interface or USB), using a known security vulnerability (CVE is Common Vulnerabilities and Exposures), and saving files to a particular location on the device |
| connection from [source network|device type] to [destination] IP address ranges speaking [protocol] in [date/time range] | A device or network made a connection to a group of IP addresses using a specific communication method during a certain time period |
| module loads of [type] on [operating system] from [path] via sessions over [protocol] | A software component (like a driver or plugin) was loaded on a specific operating system, from a certain folder or location, and it happened during a session using a particular protocol |
| hosts of [operating system] which have responded with [byte sequence] after receiving [byte sequence] via [protocol] on [port] in the last [time period] | Devices running a certain operating system replied with a specific pattern of data after receiving a certain input, using a particular protocol on a specific port, in a given time period |
| Use of [API] with [parameter] on [IaaS|PaaS|SaaS] followed by [API] which resulted in [configuration change] | Someone used a cloud service API with a specific setting, and then used another API call that changed how something was set up, possibly altering security settings, access controls, or infrastructure |
But many organisations struggle to employ TTPs as part of their detection strategies across their systems.
To use TTPs effectively, organisations need:
- comprehensive visibility across their systems
- infrastructure that allows searching and correlating activity
- skilled defenders who can build and test hypotheses based on attacker behaviour and objective
Organisations – or those who provide services to them – should not only ingest and detect IOCs but also be capable of consuming, creating, sharing, and detecting TTPs in their threat hunting. This dual approach enhances both reactive and proactive security capabilities, improving overall resilience against sophisticated adversaries.
Getting help through NCSC Assured Services
If your organisation doesn’t have the skills or capability, or otherwise requires support, the NCSC provides assurance of a range of professional services delivered by the private sector on our behalf:
- For threat hunting, the NCSC’s Cyber Incident Response providers at the Enhanced level can support you.
- For validation that your threat hunting capabilities are working as intended, use the NCSC’s new Cyber Adversary Simulation (CyAS) – announced in May 2025 – which will be opening to service providers soon.
Ollie Whitehouse
Chief Technical Officer (CTO), NCSC
Share and print this article
Written by
