Blog post

NCSC IT: Installing software updates without breaking things

Created:  08 Mar 2018
Updated:  08 Mar 2018
Author:  Andy P
Broken laptop

Software updates can be a controversial topic. We all know it’s important to install security fixes quickly after they become available, and most of us enjoy the new features that come with them. However, some organisations are reluctant to install updates before they've been fully tested, and will often disable automatic updates in order to do so.

At the NCSC, we take a different approach. We balance the security benefits of installing updates promptly, whilst also ensuring our business continues to operate once the updates are installed. This blog post explains how we do it.

Whilst the overall approach is similar on each platform, what we can technically achieve on each is different. I’ll break down what we do on a per-platform basis so we can cover the specifics.


Windows devices: operating system

Everyone in the NCSC is issued with a laptop or tablet running Windows 10 (other options are available, and we want to offer our users a choice, but we haven't prioritised doing that yet), so we need to think about how to keep Windows up to date. Traditionally in an organisation deploying Windows domains, you would use Windows Server Update Service (WSUS) to manage this. However, we don't have an on-premises network, so we use Windows Update for Business instead. This allows us to manage how the device take updates, using the public Windows Update service as a source, without needing to run our own infrastructure. Using 'Windows Update for Business', we configure our devices to install quality updates as soon as they are available, and feature updates after a short period of testing.

To ensure that forthcoming updates will not break our configuration (or cause stability problems), we have a small number of users (around 2%) enrolled on the Windows Insider programme for Business. These users receive pre-release versions of Windows on their devices, so they can spot any problems that may arise from future features. This allows us to report the problems to Microsoft, or fix them ourselves before they become a widespread issue. If the Windows Insider devices fail (or become unreliable), we have spare devices on hand in a known good state. 

All this means that even after we've tested the Windows Insider releases, we can use 'Windows Update for Business' to choose which of our users get the final update first, and confirm that it doesn't cause any additional issues.


Windows devices: firmware

As we’ve discussed extensively in blogs, keeping device firmware up to date is becoming increasingly important. Our Windows tablets support firmware updates through Windows Update, so they’re taken care of by the configuration above. However, our Windows laptops require a more bespoke approach. To ensure they’re up to date, we use the approach Mike H outlined in his blog on Automating Firmware Updates, and covered in more detail in the latest Windows 10 EUD Guidance. It explains how we use the System Center Configuration Manager (SCCM) to push firmware updates out using Custom Task Sequences, and track the install rates on our devices. Because beta programmes for firmware updates aren't generally available, we test releases on a small number of devices before we add them to SCCM.



As we mentioned in our first blog on NCSC’s IT, a lot of NCSC staff use corporately managed iPhone devices, and we manage them using our Mobile Device Management (MDM) service. iOS today doesn’t have a way to force software updates onto devices that are locked with a passcode, so we use the MDM to report device versions to the admin console, and send notifications to those users who are not installing the latest updates.

As with Windows, we want to be confident that the latest updates will not break our business, so we have a small number of users who have enrolled their main devices into the AppleSeed programme to test the latest versions of iOS before general release (the public beta programme also works for this). They can report any issues they find before they affect the rest of our user base, and can use a backup device (running the previous stable version of iOS) if their test device fails. This way we can allow (and even encourage) our users to update their iOS devices on release dates with confidence. Even though iOS now has the ability for an organisation to delay updates for up to 90 days, we have no plans on using this.

It's worth noting that it's important to install updates for associated services reasonably promptly after client updates. At the NCSC we had an issue with our MDM when it fell a couple of versions behind and didn't support the latest version of iOS, resulting in some devices appearing to fall out of compliance and being automatically wiped. We've now solved this by ensuring MDM updates are applied much more promptly.


Third-party applications and infrastructure

Following an update to our MDM in mid-2017, we can now automatically update all MDM-delivered apps on our iOS devices. This means that users no longer need to worry about installing updates manually - their devices automatically update apps when they’re connected to Wi-Fi and power. We can also use our MDM to see how effective this policy has been.

As we’ve mentioned before, a lot of our cloud services are Software as a Service, meaning that periodic updates and maintenance are taken care of by the provider. However, where we are using Infrastructure as a Service, we need to make sure our updates are installed ourselves. We do this by scheduling maintenance shortly after the manufacturers have released software updates, and because we have duplicate servers we can always fail over if the update doesn’t go as planned. 

This covers a few of the important areas to consider for updates, but not all areas. There’s plenty we’ve not covered that we’ll leave for another day, but hopefully these details will give you some ideas to think about for deploying software updates within your own organisations. We’re always reviewing all aspects of our IT management, including updates, and make changes where appropriate. If you have any suggestions of ways to improve software updates within organisations, let us know in the comments below.

Andy P

EUD Security Research Lead

Leave a comment

Was this blog post helpful?

We need your feedback to improve this content.

Yes No