Microsoft has released its post-incident report for the November 12 outage that knocked out Outlook and Exchange Online for offices across the Americas, including plenty in Southwest Florida. In plain terms: a bug Microsoft introduced in July sat unnoticed in the servers that front Microsoft 365 for more than three months. On November 12, extra traffic plus a group of servers already offline pushed those servers into the bug, and the systems designed to route around failures spread it worldwide. Email stayed down longest because Exchange Online couldn’t be moved to Microsoft’s backup gateway the way Teams and OneDrive could. No customer caused it, so the useful lessons are about how your business plans for the next one.
Key takeaways
- The cause was a hidden defect in a Windows component built only for Microsoft’s networking gateway servers. It doesn’t affect the PCs in your office.
- Three things lined up: the July bug, extra load when Teams moved onto the same gateway, and about 14% of gateway servers already out for an unrelated problem.
- Automatic failover made it worse. Within two hours, more than 80% of the gateway servers had stopped responding.
- Teams, OneDrive and SharePoint came back earlier because they could switch to a second gateway. Email couldn’t.
- Plan for a day without email the way you plan for a storm: a way to reach staff and clients, and your own backup of your mail.
What caused the outage?
Microsoft’s post-incident report, shared with Microsoft 365 administrators (a copy was posted on r/sysadmin), names three factors.
- A bug shipped in July. A fix made specifically for the gateway servers caused lock contention inside a Windows process. Under certain load it used up all the CPU and memory until the server froze. It rolled out through Microsoft’s normal staged process and ran worldwide for over three months without a problem. Microsoft’s load tests missed it because the bug is worse on newer, higher-core hardware than the lab machines they tested on.
- Extra load. Microsoft runs two independent gateway systems. About 15 hours before the outage, Teams moved from the other one onto the main gateway that also carries Exchange Online. That’s routine, but it added traffic.
- Reduced capacity. Nearly 14% of the gateway servers were out of service from an unrelated issue the day before. That alone wasn’t a problem. Combined with the Teams traffic during peak US hours, it tipped the servers into the bug.
Why did it spread everywhere?
When a server stops responding, Microsoft automatically moves its users to healthy servers. Usually that keeps an outage small. This time it moved the same load onto fewer machines, pushing each of them into the same bug, so the failure cascaded from server to server. The barriers between regions that should have stopped it also failed because of a separate issue, so it went global.
Why was email out longer than Teams?
Teams, OneDrive, SharePoint and the Office apps could switch to Microsoft’s second gateway system, so most Teams chats and meetings kept working. Microsoft says Exchange Online can’t move off this gateway because of its scale, so email had to wait for the main gateway to be repaired.
Recovery was slow because servers that came back were swamped by waiting traffic and froze again, and because a months-old change can’t simply be rolled back with a switch. Microsoft built an update that restored the earlier code, then patched and restarted servers in large groups, a region at a time.
What should a Southwest Florida business take from this?
It fits a pattern we wrote about recently in Software Keeps Breaking: several ordinary things lining up inside a system too big for any one test to catch. You plan for storms you can’t prevent. Treat outages the same way.
- Decide what you use when email is down. Teams mostly worked on November 12. Pick Teams, a group text or a phone tree ahead of time, and keep staff and key client contacts somewhere other than Outlook.
- Test your backup plans. Microsoft had two gateways and regional failover, and email still went down. A second internet line or cellular backup is only useful if you’ve confirmed it switches over.
- Back up your Microsoft 365 data separately. A separate backup of mail, OneDrive and SharePoint lets you reach important messages and files during an outage and recover them after one.
- Keep a simple change log. Microsoft’s bug took months to surface. When something breaks in your office, knowing what changed last summer makes it much faster to fix.
- Watch for fake “restore your mailbox” emails. They follow every big outage. Microsoft doesn’t ask you to sign in through a link.
Our guide from the day of the outage covers what to check once mail is flowing again.
Frequently asked questions
Does this bug affect our office computers?
No. Microsoft says the defect and the change that caused it were specific to its Microsoft 365 networking gateway servers and don’t apply to other Windows systems.
Could we have prevented the outage?
No. It happened entirely inside Microsoft’s infrastructure. What you control is how your business keeps working during an outage and how fast it recovers.
Should we move off Microsoft 365?
For most small businesses, no. It’s still more reliable than an in-office mail server, especially in an area where power and flooding are real risks. Plan for the occasional outage and keep your own backup.
Want an outage plan that’s ready before season gets busy?
We review email, file and phone continuity as part of a free consultation for businesses from Tampa to Naples, with the findings in writing. Book a free consultation or call (813) 999-0086.