Post Incident Review (PIR) – Azure Portal and other management portals – Access issues in multiple regions

We performed operations to allow management portal traffic to bypass AFD.

After verifying recovery of the previous incident, we took steps to migrate management portal traffic back completely through AFD.

Management portal availability issues occurred and impact grew.

Initial automated alerts were triggered.

Engineers were engaged and began investigating.

We confirmed the issues were related to the traffic migration and continued to investigate contributing factors.

We recovered one hosting service domain to correctly resume traffic through AFD and purged its caches to speed up recovery. Gradual recovery happened over the next 60 minutes, though other residual impact continued.

Azure Portal availability was mitigated, however other residual impact remained for other management portals. We shifted our focus to investigating these other residual failures and mitigating this impact.

We recovered an additional hosting service domain to correctly resume traffic through AFD. After reviewing our configuration and monitoring telemetry, we confirmed mitigation.