Post Incident Review (PIR) – Service management operations failures in East US 2

Customer impact began in AZ03.

Monitoring systems observed a spike in allocation failure rates for Virtual Machines, triggering an alert and prompting teams to investigate.

Initial targeted communications sent to customers via Azure Service Health. Automated messaging gradually expanded as impact increased, and additional service alerts were triggered.

We began manual mitigation efforts in AZ03 by restarting critical service components to restore functionality, rerouting workloads from affected infrastructure, and initiated multiple recovery cycles for the impacted backend service.

Platform services in AZ02 began to throttle requests due to the accumulation of excessive load, following customer deployments being automatically redirected from AZ03, in addition to normal customer load in AZ02.

Platform-initiated retries caused excessive load in AZ02, leading to increased failure rates.

We stopped new VM deployments being allocated in AZ03.

The reduction in load in AZ03 enabled AZ03 to recover. Customer success rates for operations on VMs that were already deployed into AZ03 returned to normal levels.

After verifying that platform-initiated retries had sufficiently drained, we re-enabled new VM deployments to AZ03. We then stopped all new VM deployments in AZ02.

We continued to apply additional mitigations to AZ02 platform services with backlogs of retry requests. These included applying more aggressive throttling, draining existing backlogs, restarting services, and applying additional load management strategies.

Customer success rates for operations on VMs already deployed into AZ02 started increasing.

Broad targeted messaging sent to all customers with Virtual Machines in East US 2.

We started gradually re-enabling traffic in AZ02.

The first update was posted on the public Azure Status page.

After a period of monitoring to validate the health of services, we were confident that the control plane service was restored.

Remaining downstream services reported recovery, following backlog drainage and retry stabilization. Customer impact confirmed mitigated.