Vulnerability management

When vulnerabilities are being exploited by automated means, every hour that passes without the update being applied increases the chance of a compromise. As such, organisations should have a rapid response pathway or policy for updates. This policy has to account not just for well-supported corporate systems, but also for shadow IT as an attacker will enter through any route available.

In any organisation bigger than a few people, preparation is crucial to ensure that critical updates can be identified and applied without breaking the business. In larger organisations this process will likely be owned and maintained by the vulnerability management team or wider security operations. However, creating a policy where one does not exist will require strong buy-in from IT operations or admin staff, and so is likely to require support from senior leadership. Additionally, the rapid response process will need to be reflected in supplier contracts. Organisations need to ensure that critical suppliers or managed service providers are contractually liable to rapidly mitigate vulnerabilities in internet-accessible systems, when these vulnerabilities are being exploited in the wild.

Updates will normally be applied by an IT admin or operations team (rather than the security team) and so plans for a rapid response must include such teams. In smaller organisations that don’t have formal IT outage or incident management processes, a simple agreement may suffice with IT staff, committing them to prioritising applying updates when the situation requires it.

In medium and large organisations, there will likely be an established process to handle IT outages and incidents that triages outages / incidents into different priorities, with the highest priority responses, known as as Priority 1 or P1 incidents. These are reserved for business-breaking outages and will require extensive, rapid response, often with retroactive change requests.

Security teams should establish formal agreements with IT incident owners to ensure that rapid vulnerability remediation is assigned the same priority level as a total system compromise.

Applying updates can cause friction between security and IT teams, so security leadership (such as the CISO) should be consulted as to how best manage any potential issues. Security teams will likely need to prove (through their assessment methodology and process) that this pathway will only be triggered in genuinely time-critical situations, where delayed action is effectively guaranteed to lead to a compromise.

Organisations should be monitoring vulnerability feeds as part of business-as-usual activities, most likely through the general vulnerability management process or a procured service. Vendors will also often send notices of critical updates to registered users. Such updates should be covered by an update by default policy, and ideally installed automatically.

However, mass exploitation using automated means can scale incredibly rapidly. For example, if a threat actor is able to create an exploit and then start attacking systems, other groups quickly copy the exploitation technique. Responding quickly at this this point is crucial, and responses need to include ensuring there are not vulnerable instances on shadow IT or other systems not well-known to the IT and security teams.

Timely awareness of vulnerabilities gives the organisation as much time to respond as possible. A process for ensuring such awareness should be created (if not already existing) and maintained by the appropriate staff. The process should include how news of breaking vulnerabilities or exploitation will be monitored and assessed in a way that is effective, but does not overly burden those responsible.

Critical vulnerabilities and ‘in the wild’ exploitation are likely to be reported in the following places:

• within vendors advisories, which will be on their official website (this will include information about the vulnerability and any mitigations, and may also include information regarding exploitation, indicators of compromise, scripts and other support)
• within security vendor websites, particularly those in the vulnerability assessment and management space
• on the websites and social media feeds of national cyber agencies such as the NCSC, CISA, NSA, ASD, CCCS and NCSC-NZ
• within updates to the CISA Known Exploited Vulnerabilities (KEV) catalogue
• in the cyber / IT press and websites, or on associated social media

When critical updates are announced in software or dependencies used by the organisation, CISA’s Stakeholder-Specific Vulnerability Categorization (SSVC) decision tree should be used to assess the need for action. For example, if there is proof-of-concept code available for the vulnerability (and successful exploitation has the potential to cause a high level of harm to your organisation), then you should remediate the vulnerability sooner than your business-as-usual timelines would require.

The table below highlights the compressed timelines for installing updates when a vulnerability on the CISA Known Exploited Vulnerability Catalog (CISA KEV) is being actively exploited in a business-critical system. To help assess your level of exposure, we also suggest you conduct a full investigation through an assured Cyber Incident Response (CIR) provider. Where this this isn’t possible, the affected system should be erased/destroyed and rebuilt as new before applying the latest updates and hardening.

In organisations with vulnerability management processes, there will be a register of known services from inventory and vulnerability scan activities. Identification of vulnerabilities in officially supported products or other software should therefore be possible through the normal vulnerability management process.

Re-scanning hosts/ports that are believed to host the affected software with an updated vulnerability scanner should identify whether you are affected. However, if it’s worth checking that the vulnerability scanner has implemented detections for that particular vulnerability (or where applicable, that your outsourced solution is doing so). It may take a few hours or more for your scanner or provider to include detections.

The NCSC has worked with multiple organisations who have had security incidents on their systems that their security and IT teams were unaware of. When there is mass exploitation of critical vulnerabilities, organisations are recommended to confirm they are not affected beyond the known systems that may be included in normal vulnerability scanning. This can include shadow IT, developer environments, contractors or other embedded staff.

Security teams should consider non-traditional data sources when trying to identify if they might be running software or appliances. This can include:

• Email security services or gateways which allow security teams to search for emails from specific domains. Searching for emails from the affected vendor can identify mailboxes and users who might be using the product and can be queried.
• Looking at web proxy logs on perimeters for connections to the vendor site.
• Actively monitoring vulnerability disclosure and reporting portals for reports of vulnerable systems discovered by security researchers.

A more challenging scenario to diagnose the is when a vulnerability is not within a specific product or software, but within a library or other package that is integrated into another piece of software, such as was seen during the Log4J incident.

If your organisation produces products and software, then hopefully automated compilation and build pipelines are used and so artifacts like software bills of material (SBOMs) are produced and can be queried for the affected package. Where this is not the case or is suspected to not be used by all those developing software, then searching code repositories can identify usage of specific libraries or packages.

The non-traditional data sources listed above can also help identify if you are using software containing the affected package.

Concerns about affected libraries/packages being used by third-party software should be addressed through the above processes concerning software generally, i.e. keeping an eye on vendor update releases, should they discover they are affected.

Assess whether attackers can access the service

A key discriminator in how urgently you need to remediate a vulnerability once it's confirmed is its exposure. If you have followed the above guidance, then by this stage:

• the vulnerability is known to be critical
• you have affected systems
• there is active exploitation of the vulnerability

The final stage of triage is then how reachable the vulnerable system is. That is, can an attacker get to it. Your business-as-usual vulnerability management process may already have confirmed this through vulnerability scanning from the internet. However if the vulnerable systems have been discovered as part of the response, you should confirm they are reachable from the internet. It is important to confirm that you own the system before doing so however, particularly with suspected shadow IT.

Whilst security teams might want to study exploit code that has been released, they should not use that code to confirm vulnerability without expert assessment that the code does not contain backdoors or other malicious behaviour. There is a long history in the security world of ‘exploit code’ that instead attacks the person running it.

Trigger the IT outage / incident process

Formal integration with the IT outage / incident process, where it exists, is by far the most effective way to get the issue fixed quickly and safely as the process should include relevant stakeholders and be able to handle ambiguity. Once the issue has been confirmed as critical on an exposed system(s) owned by the organisation (and with ‘in the wild’ exploitation), the threshold for declaring an incident should have been met.

In organisations where a formal incident process doesn’t exist, remediation will be more ad-hoc, and so there are some key response priority actions that need to be followed.

1. Read the vendor advisory and assess your level of compromise. Has the vendor (or a third party) released instructions for confirming compromise?
2. If possible isolate the affected system(s) and replace with a new, fully up-to-date system (NOTE: this may cause service outage). For example, temporarily disable access to the service with upstream firewalls, disable the vulnerable component(s) or restrict access to only the organisation's IP range.
3. Fully investigate for evidence of compromise. Updating during widespread automated exploitation is a race against time, and so while IT teams are applying updates or other fixes, security staff (ideally an assured Cyber Incident Response provider) should be assessing whether there are signs of compromise. This is best done with understanding of the vulnerability and the exploits for it. Examples include:
   1. Does exploitation involve outbound communications to a domain or IP address? Search DNS and traffic monitoring for the IP address or domain.
   2. What does outbound traffic from the vulnerable system normally look like?
   3. What sort of systems and ports internally does it usually connect to?
   4. Have there been any configuration changes since exploitation was known to have started? Where this this isn’t possible, the affected system should be erased/destroyed and rebuilt as new.
4. Report the compromise - If you believe you have been compromised, and are in the UK, you should report it. You can also report the compromise to the vendor to assist their investigation.
5. Update to the latest version of the affected product. In the majority of cases, the update should be possible to apply with no issues. If the update is more complex, organisations should inspect update notes, check online forums, or contact their support function with the vendor.
6. Update systems that are not business-critical first – to test the process, follow the organisations normal update rings (i.e. staging the update across different groups of servers), but with a greatly accelerated timeline (perhaps only minutes between rings).
7. Apply any appropriate security hardening.
8. Re-enable/reintroduce the affected system(s).
9. Perform continuous threat hunting activities.

Once the vulnerable systems have been mitigated, ideally through updating, there remain steps to take to make the organisation more resilient. An important adage in security is ‘never let a crisis go to waste’, and the organisational interest in preventing a ‘near miss’ by responding to an urgent updating scenario is likely to be usable to accelerate related security efforts.

An exploitable, high impact vulnerability in a service demonstrates to security researchers and threat actors that there are likely more vulnerabilities to be found in the same product. As such, once there has been a vulnerability in a product that’s seen a wave of attacks, there is a greater chance there will be future vulnerabilities found and mass-exploited, a process known as ‘vulnerability swarming’. As such, once you have handled the current issue(s), you have to work on the assumption it will happen again. If this happens, you should also be ready to deploy additional vendor updates in the days immediately following the disclosure of a new vulnerability. This shouldn’t stop you rolling out the first update.

Is the system worth the risk?

Studying the types of software or appliance that see mass exploitation identifies a few product categories that are more often affected, notably VPNs, file transfer solutions, and remote access products. In all these cases, there are more modern approaches to the problems being solved, and so the organisation may now be more willing to consider removing the system or replacing it.

If the system can’t be removed or replaced, can you better secure it?  Vulnerabilities that see mass exploitation are often in functionality that is not the core functionality of the software or appliance itself, but often is enabled by default. To greatly reduce the chance of being affected by future vulnerabilities, all unnecessary functionality should be disabled or otherwise blocked (for example by firewalling). The system should also be hardened to exploitation through defence in depth, for example by placing network or identity restrictions on what the system can access and creating alerts for suspicious activity.

It is almost a certainty that the organisation will need to respond to a future critical vulnerability that is being exploited, so a key after-action step is to consider what would have made the process easier and to work to have it in place for next time. This can include access to data sources to identify shadow IT, ensuring all teams use the same code repositories, improving relationships with IT operations staff, etc.