Blog post

NCSC IT: Installing software updates without breaking things

Created:  08 Mar 2018
Updated:  08 Mar 2018
Author:  Andy P
Broken laptop

Software updates can be a controversial topic. We all know it’s important to install security fixes quickly after they become available, and most of us enjoy the new features that come with them. However, some organisations are reluctant to install updates before they've been fully tested, and will often disable automatic updates in order to do so.

At the NCSC, we take a different approach. We balance the security benefits of installing updates promptly, whilst also ensuring our business continues to operate once the updates are installed. This blog post explains how we do it.

Whilst the overall approach is similar on each platform, what we can technically achieve on each is different. I’ll break down what we do on a per-platform basis so we can cover the specifics.

 

Windows devices: operating system

Everyone in the NCSC is issued with a laptop or tablet running Windows 10 (other options are available, and we want to offer our users a choice, but we haven't prioritised doing that yet), so we need to think about how to keep Windows up to date. Traditionally in an organisation deploying Windows domains, you would use Windows Server Update Service (WSUS) to manage this. However, we don't have an on-premises network, so we use Windows Update for Business instead. This allows us to manage how the device take updates, using the public Windows Update service as a source, without needing to run our own infrastructure. Using 'Windows Update for Business', we configure our devices to install quality updates as soon as they are available, and feature updates after a short period of testing.

To ensure that forthcoming updates will not break our configuration (or cause stability problems), we have a small number of users (around 2%) enrolled on the Windows Insider programme for Business. These users receive pre-release versions of Windows on their devices, so they can spot any problems that may arise from future features. This allows us to report the problems to Microsoft, or fix them ourselves before they become a widespread issue. If the Windows Insider devices fail (or become unreliable), we have spare devices on hand in a known good state. 

All this means that even after we've tested the Windows Insider releases, we can use 'Windows Update for Business' to choose which of our users get the final update first, and confirm that it doesn't cause any additional issues.

 

Windows devices: firmware

As we’ve discussed extensively in blogs, keeping device firmware up to date is becoming increasingly important. Our Windows tablets support firmware updates through Windows Update, so they’re taken care of by the configuration above. However, our Windows laptops require a more bespoke approach. To ensure they’re up to date, we use the approach Mike H outlined in his blog on Automating Firmware Updates, and covered in more detail in the latest Windows 10 EUD Guidance. It explains how we use the System Center Configuration Manager (SCCM) to push firmware updates out using Custom Task Sequences, and track the install rates on our devices. Because beta programmes for firmware updates aren't generally available, we test releases on a small number of devices before we add them to SCCM.

 

iPhones

As we mentioned in our first blog on NCSC’s IT, a lot of NCSC staff use corporately managed iPhone devices, and we manage them using our Mobile Device Management (MDM) service. iOS today doesn’t have a way to force software updates onto devices that are locked with a passcode, so we use the MDM to report device versions to the admin console, and send notifications to those users who are not installing the latest updates.

As with Windows, we want to be confident that the latest updates will not break our business, so we have a small number of users who have enrolled their main devices into the AppleSeed programme to test the latest versions of iOS before general release (the public beta programme also works for this). They can report any issues they find before they affect the rest of our user base, and can use a backup device (running the previous stable version of iOS) if their test device fails. This way we can allow (and even encourage) our users to update their iOS devices on release dates with confidence. Even though iOS now has the ability for an organisation to delay updates for up to 90 days, we have no plans on using this.

It's worth noting that it's important to install updates for associated services reasonably promptly after client updates. At the NCSC we had an issue with our MDM when it fell a couple of versions behind and didn't support the latest version of iOS, resulting in some devices appearing to fall out of compliance and being automatically wiped. We've now solved this by ensuring MDM updates are applied much more promptly.

 

Third-party applications and infrastructure

Following an update to our MDM in mid-2017, we can now automatically update all MDM-delivered apps on our iOS devices. This means that users no longer need to worry about installing updates manually - their devices automatically update apps when they’re connected to Wi-Fi and power. We can also use our MDM to see how effective this policy has been.

As we’ve mentioned before, a lot of our cloud services are Software as a Service, meaning that periodic updates and maintenance are taken care of by the provider. However, where we are using Infrastructure as a Service, we need to make sure our updates are installed ourselves. We do this by scheduling maintenance shortly after the manufacturers have released software updates, and because we have duplicate servers we can always fail over if the update doesn’t go as planned. 

This covers a few of the important areas to consider for updates, but not all areas. There’s plenty we’ve not covered that we’ll leave for another day, but hopefully these details will give you some ideas to think about for deploying software updates within your own organisations. We’re always reviewing all aspects of our IT management, including updates, and make changes where appropriate. If you have any suggestions of ways to improve software updates within organisations, let us know in the comments below.


Andy P

EUD Security Research Lead

6 comments

John Rossetti - 19 Mar 2018
It's a great shame this sensible approach cannot be implemented for the (slightly) more experienced home user. As someone who has a couple of pc's for small but critical business use at home (W7 & W10) I am very torn between updating anything, or not, because of the number of very poor quality updates coming out of Redmond.
I now totally rely on Ask Woody https://www.askwoody.com/ as his advice, which is a real pain and not good use of my time.
Even with this amount of caution I have had my brand new (Dec) W10 pc “bricked” twice by M$ when “forced” to update to v1709
I wish there was a corporate service update advice service available.
Darren G - 20 Apr 2018
I'm intrigued by the "we don't have an on-premises network" statement - No network, at all? - An interesting approach. Possibly covered in other blog posts but I'd be keen to understand more about that decision and what drove it. I can see potential benefits but I'd like to understand how you manage and protect devices (assuming more than simply Windows Update for Business), how do you work together as an organisation, collaborate etc. Is everything Cloud based? You mentioned SCCM which suggests there is a network of sorts. Love to hear more on that.
Andy P - 25 Apr 2018
Hi Darren. We have a small amount of equipment to manage on-premises equipment (e.g. the Wi-Fi network), but really this is only to provide our devices with a route to the internet. Once they have a connection, the always-on VPN kicks in and connects to cloud infrastructure providing access to network services.

We use some IaaS for hosting SCCM/DC/AD etc. This decision was made to provide as much flexibility and reliability as possible; our users in remote sites have no requirement on any other site being operational in order to have working IT.

You can read more about the architecture in other NCSC IT blogs: https://www.ncsc.gov.uk/search?keyword=NCSC%20IT&f%5B0%5D=type%3Ablog”
Mark - 20 Apr 2018
This is useful, but not entirely practical for our small business. We currently all run 1 year old PCs with Windows 10 and an Office 365 subscription. Several times over recent months Windows updates have stopped Office from working, causing significant disruption and leading to us missing key deadlines. We have spent many hours on the phone to MS tech support getting things fixed. Their advice to us? Disable automatic updates! I am concerned about our security, but frankly MS was doing more damage to our business than any hacker ever has. I wish someone could hold MS to account for this sort of thing - they just seem to push out poorly tested updates and damn the consequences.
Matt - 24 Apr 2018
Interesting article - how many Staff does this cover in NCSC and what number of apps on Windows 10 are managed?
Andy P - 03 May 2018
Hi Matt. We currently have a little over 1,000 Windows 10 devices in SCCM - A handful of those will be test devices, and the rest should be devices issued to users.

We have 50 apps that are currently deployed via the application catalogue. They aren’t all published to everyone, as some have licensing costs / implications and some are fairly niche. However the idea was for free useful stuff to be published to everyone. For example - I only see 23 in the software center.

Leave a comment

Was this blog post helpful?

We need your feedback to improve this content.

Yes No