Outlook, Teams, and Microsoft 365 customers were locked out of their accounts for two days last week following an extended outage, leaving millions without access to email and causing business systems to struggle.d Outages are typically a few hours long, but last week’s was notable for its length and the breadth of systems taken offline.
As PCMag reports, in a post-mortem posted to X on Monday, Microsoft’s executive for Copilot, Agents, and Platform, Charles Lamanna, said the incident was due to “three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services.”
Factor #1 – Defect From a Fix for Microsoft 365
The first factor stemmed from a defect introduced in July as part of a fix for Microsoft 365 networking gateway servers. Microsoft says the software, including this hidden defect, was “saturated globally over several weeks” without any impact.
“Our normal load testing did not detect the latent issue because [of] the different traffic and SKU mix used in our test labs,” says Lamanna. The issue is “worsened on machines with more cores and newer hardware,” but Microsoft’s own load testing focuses on “weaker hardware SKUs with fewer cores.”
Factor #2 – Surge in Load Linked to Microsoft Teams
The second factor was a surge in load on its networking gateways, linked to a minor issue with Microsoft Teams on Nov. 11. That prompted Teams to switch to an alternative networking gateway, increasing load and changing overall traffic patterns. “This type of failover is part of normal operations and is integral to standard Microsoft resiliency efforts,” Lamanna said.
Factor #3 – Unrelated Issue???
The third factor is what Lamanna calls an “unrelated issue from the previous day,” in which 14% of Microsoft’s server capacity for its networking gateway was left out of service. The second and third factors combined meant that the networking gateway encountered the defect first introduced in July.
Result – Chaos, Followed by Long Wait & Lack of Service
This led to a buildup of requests and ultimately caused servers to stop responding, all during what Microsoft calls the “peak window of traffic in the United States.” Microsoft’s automated systems then rerouted requests, causing the defect to recur across its network and eventually affecting more than 80% of its servers.
“Our normally automated recovery processes were not designed to restore service from this broad unresponsive state,” Lamanna said.
To solve the problems, engineers developed a fix and took each server offline to apply it. Engineers then carefully reactivated each server, resulting in long wait times and therefore a lack of service for Microsoft customers.
The thread says Microsoft plans to share a full post-incident review later this week, although an exact timeframe isn’t yet available. Lamanna said, “We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.”
—
Photo Credit: Mongta Studio / Shutterstock.com