Two of the biggest technology failures of the year happened within a day of each other last week, and both appear to have started with a change. That makes them as much a story about change management as about recovery.
Microsoft’s outage began on the evening of Thursday 12 November and ran well into Friday, with full service not back until Saturday morning. Microsoft has said that a process that sets up secure connections was becoming overwhelmed, which caused affected servers to stop handling requests, and it began rolling back a recent network configuration change that coincided with the start of the disruption. It hasn’t yet confirmed a root cause, but it has been clear that this was not a cyber attack.
Later on Friday, the US Federal Aviation Administration ordered a nationwide ground stop that lasted several hours. The root cause is still under investigation, but according to the US Department of Transportation, a router replaced during routine work by a contractor to switch old copper lines to fibre started flooding other routers with more than 1,000 connection attempts every four to five seconds, which disrupted communications with aircraft and radar feeds.
Plenty has already been written about the Microsoft outage, much of it by people who spent Thursday night and Friday living with it. Most of that commentary covers business continuity and disaster recovery, supplier dependency and the need to test your workarounds. All of it is valid. What I have seen far less of is any discussion of the change itself, and that seems to me to be where both incidents actually began.
Two outages, one pattern
Strip away the technology and the two incidents look very similar. Someone made a change that was expected to be routine. The change appears to have consumed more of something than the system could cope with, whether that was the capacity to handle secure connections at Microsoft or the capacity of the network in the FAA’s case. The impact then spread well beyond the thing that had been changed.
In the UK, the effects of the Microsoft outage were felt in places that had nothing to do with the change and no say in it. Hospitals, GP practices and community health services were disrupted, with some operations cancelled, some staff falling back on paper records and urgent referrals delayed. The National Pharmacy Association warned that some prescriptions could be held up because pharmacies couldn’t reach GP practices by NHS email. Schools and the Department for Education were affected too.
None of those organisations could have prevented Microsoft’s change. But every one of them makes changes of its own, every week, and the same pattern can play out on a much smaller scale.
The kettle and the substation
Change management is usually thought of as an IT discipline, with change requests, approval boards and rollback plans. Some of the most damaging changes, though, never go anywhere near a change board, because nobody thinks of them as changes at all.
Back in 2004, at a previous employer, there was a call centre that kept suffering power problems. Whenever someone plugged in a kettle or a toaster, the power problems started again, because the building was already running at capacity. The organisation had recognised the risk and built an overflow facility, and I was proudly shown around it. Everything stood ready, right down to staff family photos on the desks. It had failed on day one because there was insufficient power to run it, and they were now waiting for a new substation to be built.
The second example comes from a client of a different employer, in the broadcasting sector. Over time, production teams had bought better and brighter lights, better cameras and other kit, each purchase perfectly sensible in isolation. Nobody was adding it all up. By the time anyone looked, the building was running at 104% of the capacity of its electrical panel. It was not the only example of change management failing in that organisation, but it was by far the starkest.
In neither case did anyone do anything obviously wrong. A kettle is not a change request, and nor is a new lighting rig bought out of a departmental budget. The overflow site was built precisely because someone had thought about continuity. What was missing was anyone asking what the change would do to everything else around it, and in particular what it would consume.
Why capacity is so easy to miss
Most change processes ask whether a change will work. Far fewer ask what it will use up. Power, network bandwidth, connection limits, cooling, floor space, people’s time and the attention of the service desk are all finite, and most of them are shared. Each individual change takes only a little, which is exactly why the total creeps up unnoticed until one more small change tips it over.
Backup and overflow arrangements are particularly exposed. They tend to be designed once, signed off and then left alone, while the live environment they are meant to replace keeps changing around them. An overflow site sized for last year’s operation may not cope with this year’s. A failover route that was tested before a network upgrade may behave quite differently afterwards.
Where business continuity fits in
ISO 22301 already expects organisations to review their business impact analysis and risk assessment when there are significant changes, not just at planned intervals. The difficulty is deciding what counts as significant. A data centre migration obviously does. A copper to fibre replacement at one remote site probably wouldn’t feel like it, and nor would a new set of studio lights. Yet those are exactly the kinds of change that caught people out here.
For organisations working to ISO 27001, the 2022 version of Annex A includes controls for both change management and capacity management. In many organisations the two sit with different teams, and the business continuity lead is often not involved in either.
That is the gap I would most like to see closed. The person responsible for continuity does not need to approve every change, which would be impractical and slow everything down. But they should be able to see what is changing, and the change process should prompt a few questions that connect each change back to the organisation’s critical activities.
Questions To add to your change Management process
None of this needs a new framework. A handful of questions, asked consistently, would have flagged both of my examples long before they became incidents.
What will this change consume, and how much headroom is left? That applies to power and network capacity as much as to server resources, and it means somebody has to know what the headroom actually is.
What depends on the thing being changed, and does it support any of our critical activities? If it does, the continuity lead should know about it and the BIA may need revisiting.
Does the change affect our fallback arrangements as well as the live service? If the live environment has grown, the backup needs to grow with it, and it needs testing under realistic load rather than simply being switched on to see if the lights come on.
How will we know quickly if it is misbehaving, and how fast can we back it out? Microsoft rolled back the change it suspected within hours, but full recovery still took the best part of two days.
And finally, who is adding up all the small changes? In both of my examples, no single decision was the problem. The problem was that nobody owned the total.
Lessons identified, not yet learned
Most organisations cannot influence how Microsoft or an air traffic control contractor manage their changes, although it is a perfectly reasonable question to ask key suppliers about. What every organisation can control is its own change process, and whether that process treats capacity and continuity as part of the decision rather than something to sort out afterwards.
The commentary on last week’s outages has rightly focused on what to do when a critical service goes down. The less comfortable question is how many of our own incidents start with a change that we approved, or that nobody thought needed approving at all.



