Vulnerability management

Installing the latest updates is critical to ensuring the security of your estate. You should put in place a policy to update by default, where you always apply software updates as soon as possible, and ideally automatically. This should be at the core of your update management process as the desired standard for systems, but it may not apply in some circumstances, such as for safety-critical systems or operational technology.

Different vendors publish different types of software updates which can be confusing and which come with their own complications. Issues include:

• Some separate the security updates from feature updates, while others combine them.
• Sometimes you can only install the latest update if you have installed a previous one. Multi-version upgrades are more likely to have unintended side effects than single updates.
• Vendors may publish advisories for some vulnerabilities, for example when attackers are known to be actively exploiting them, but also ‘silently’ update others, without public acknowledgement. Missing an update could mean you also miss these silent updates, heightening the risk to your organisation.

For these reasons, the NCSC recommends you install all updates as soon as possible. You can stay ahead of this by giving yourself time to perform controlled updates, rather than waiting until you are forced and you urgently need to update to the latest version to mitigate a new vulnerability.

You should also ensure the updates come from trusted sources, usually the vendor support website. You may be able to confirm the hash or checksum before installation.

Testing

Vendors carry out their own quality assurance testing of these updates and once they release them, you should also test on your own systems. This doesn’t have to slow down the rollout, as you can carry it out gradually, for example in a phased/staged rollout across your estate, or using a canary deployment model to a subset of users.

A phased rollout also allows you to carry out ‘live testing’ of the estate against the security update. Testing in the actual system, rather than a test or pre-production system, will catch real-world issues. In live testing, the rollout can be paused or rolled back if an issue is identified.

Things to consider

To make update management and roll out easier, you should consider the following:

• As a system owner, set up your communications preferences so that you receive the latest updates as soon as they are available. You should also make sure you have the required licences.
• It's better to stay within the application development rules of the platform – don’t try and ‘reinvent the wheel’. As an example, applications that are specially packaged are more likely to experience issues than a package from a platform application store, or one that uses the platform’s native installer.
• Some cloud services that support infrastructure as a service (IaaS) include tools to help with vulnerability management and implementing updates. This usually includes building your deployed IaaS on a supported operating system (OS) variant, so you should use a supported OS that has a cloud platform with an auto-update service.
• Employ infrastructure as code (IaC). The most reliable way to automate is to replace the running image or code with the updated version, rather than having to update it. Your organisation can reduce the chance of running into issues by ensuring that newly developed systems are developed in a way that allows easy testing, updating and rolling back.
• Automation is key to easing the burden. For example, you can automate the deployment of updates when scanning identifies that any are missing.

If you are using removable media, such as a USB stick, to transfer the update from the internet to another system, the removable media should be checked for viruses before installing.
 

Best-practice timescales

To help you plan your rollout process, we list our definitions of best practice below. These timescales can also help your organisation develop contracts to achieve best practice. Our timeframes are consistent with vendor best-practice advice and our own Cyber Essentials certification. They apply to all updates, regardless of the vulnerability severity.

Your aim should be to reduce the timeframe to as small a window as possible. When a vulnerability is fixed, attackers will often study the vulnerability and attempt to write exploits for it. This can lead to a race between network defenders who are updating and attackers who are looking to reach those who haven’t yet installed the update.

To help meet the timeframe, you can make use of the range of vendor tools and services which enable automatic updates and help manage rollouts. This also makes it easy to pause or roll back an update if you do experience an issue. Note that where you are using default configurations of applications or operating systems, a phased rollout may not be necessary.

For business-critical systems, you should balance the timescales below against system availability.

Updating when exploitation is rife

The above timelines are for business-as-usual updates, but there will be times when a vulnerability is discovered and attackers are scanning or attacking the internet at scale to find victims before they update. In these cases, the timelines above are too long and it is essential to speed up the update process.

Priority actions

If you need to respond to a specific vulnerability that is being actively exploited, you should refer to the vendor's product advisory. If there is no advisory, or little information available, you can use these generic immediate response priority actions:

1. Assess your level of compromise.
2. If possible, isolate the affected system(s) and replace with a new, fully up-to-date system. Note this may cause service outage.
3. Fully investigate for evidence of compromise.
   1. If possible, use an assured Cyber Incident Response provider
   2. Where this isn't possible, the affected system should be erased/destroyed and rebuilt as new. 
4. If you believe you have been compromised, and are in the UK, you should report it. You can also report the compromise to the vendor to assist in their investigation.
5. Update to the latest version of the affected product.
6. Apply any appropriate security hardening.
7. Re-enable/reintroduce the affected systems.
8. Perform continuous threat hunting activities.

It’s often best to do this using your existing internal processes for managing internal IT incidents such as outages. This is because the necessary governance processes, such as on-call staff rotas, should already be in place. Raising an IT incident shouldn’t be taken lightly, and security teams should agree with IT staff when this is appropriate, and how to activate the process.

If this happens, you should also be ready to deploy additional vendor updates in the days immediately following the discovery of a new vulnerability. This shouldn’t stop you rolling out the first update.

Note that if a vulnerability affecting an internet-facing service is being actively exploited in the wild, you should investigate your exposure and check for signs of compromise before applying any update, even if the exposure was brief. It is still possible to be compromised even after applying updates if successful exploitation occurred during the exposure window.

Sources of information include:

• The vendor advisory will include information about the vulnerability and any mitigations. It may also include indicators of compromise, scripts and other support.
• CISA’s Known Exploited Vulnerabilities Catalog.