Post Incident Review (PIR) – Virtual Machines, Managed identities for Azure resources, and dependent services – Service management issues in multiple regions

Customer impact began, triggered by the periodic remediation workflow starting.

Internal service monitoring detected a subset of regions having an increasing number of control plane failures.

Correlated issues across multiple regions.

Service monitoring detected failure rates exceeding failure limit thresholds.

We began collaborating to devise a mitigation solution and investigate the underlying factors.

We applied a primary proposed mitigation and validated that it was successful on a test instance.

We identified and disabled the remediation workflow and stopped any ongoing activity, so it would not impact additional storage accounts.

Began broader mitigation to impacted storage accounts. Customers saw improvements as work progressed.

Storage accounts for a few high volume VM extensions that utilize managed identity finished their re-enablement in East US.

Unexpectedly, customer impact increased first in East US and cascaded to West US with retries, as a critical managed identity service degraded under recovery load.

Automated alerting identified availability impact to Managed Identity services in East US. Engineers quickly recognized that the service was overloaded and began to scale out.

All extension hosting storage accounts had been reenabled, mitigating this impact in all regions other than East US and West US.

Initial scale outs of managed identity service infrastructure scale out completed, but the new resources were still unable to handle the traffic volume due to the increasing backlog of retried requests.

A second, larger set of managed identity service scale outs completed. Once again, the capacity was unable to handle the volume of backlogs and retries.

Reviewed additional data and monitored downstream services to ensure that all mitigations were in place for all impacted storage accounts.

To recover infrastructure capacity for the managed identity service, we began rolling out a change to remove all traffic so that the infrastructure could be repaired without load.

After infrastructure nodes recovered, we began gradually ramping traffic to them, allowing backlogged identity operations to begin to process safely.

Backlogged operations completed, and services returned to normal operating levels. We concluded our monitoring and confirmed that all customer impact had been mitigated.