Microsoft Says Maintenance Bug Caused Azure and Microsoft 365 Outage
Summary
Microsoft said a bug in its automated maintenance system removed IP routes from too many devices, disrupting Azure and Microsoft 365 services in the West US region. It rolled back the change and reported full recovery by 3:41 PM ET.
Key points
- Microsoft attributed the outage to a bug that incorrectly included extra network devices in routine maintenance, removing more IP routes than intended.
- The disruption began at 10:44 AM ET and affected Azure and Microsoft 365 services connected to the West US region.
- Affected services included SharePoint Online, Teams, OneDrive, Azure Kubernetes Service, Microsoft Defender, and Microsoft Sentinel.
- Some Defender customers experienced delays from Defender Experts, and investigations, workflows, and remediation actions could fail.
- Microsoft said engineers traced the route removals to the maintenance activity and began rolling back the change at 1:45 PM ET.
- The rollback finished at 2:26 PM ET; Microsoft reported all affected services recovered by 3:41 PM ET.
- Microsoft said it is reviewing its maintenance safety checks and automation and plans to publish a final incident review.
Article Details
- Event Type
- Cloud service outage caused by a bug in Microsoft’s automated network maintenance request system.
- Impact
- The bug removed IP routes from more network devices than intended during maintenance in Microsoft’s West US region, disrupting connectivity and causing failures, delays, or degraded access across Azure and Microsoft 365 services. Microsoft reported that all affected services had recovered by 3:41 PM ET after it rolled back the change.
Vendors
Products
AzureMicrosoft said a bug in its automated maintenance request system mistakenly removed IP routes from too many devices, causing a major outage that disrupted Azure and Microsoft 365 services across the West US region.Azure App ServiceAzure services such as Azure App Service, Azure Kubernetes Service, and Microsoft Graph also experienced failures or delays.Azure Kubernetes ServiceAzure services such as Azure App Service, Azure Kubernetes Service, and Microsoft Graph also experienced failures or delays.Microsoft 365 Admin CenterSharePoint Online, Microsoft Teams, OneDrive, and the Microsoft 365 Admin Center were among the impacted services.Microsoft 365 Apps for EnterpriseMicrosoft said a bug in its automated maintenance request system mistakenly removed IP routes from too many devices, causing a major outage that disrupted Azure and Microsoft 365 services across the West US region.Microsoft Defender Antimalware PlatformThe incident affected services including SharePoint Online, Microsoft Teams, OneDrive, Microsoft Defender, and others, and Microsoft later rolled back the change and said it will publish a full review.Microsoft GraphAzure services such as Azure App Service, Azure Kubernetes Service, and Microsoft Graph also experienced failures or delays.Microsoft OneDriveThe incident affected services including SharePoint Online, Microsoft Teams, OneDrive, Microsoft Defender, and others, and Microsoft later rolled back the change and said it will publish a full review.Microsoft TeamsThe incident affected services including SharePoint Online, Microsoft Teams, OneDrive, Microsoft Defender, and others, and Microsoft later rolled back the change and said it will publish a full review.SharePoint OnlineThe incident affected services including SharePoint Online, Microsoft Teams, OneDrive, Microsoft Defender, and others, and Microsoft later rolled back the change and said it will publish a full review.