Post Incident Review (PIR) – Azure OpenAI Service – Errors in multiple regions

Initial customer impact started, as cleanup operations began across regions, gradually increasing as more regions became unhealthy.

Service monitoring detected failure rates going above defined thresholds, alerting our engineers who began an investigation.

We determined there was a multi-region issue unfolding by correlating alerts, service metrics, and telemetry dashboards. It took 30-40 minutes for the delete to complete, so each region (and even each endpoint in each region) went offline at different times.

We identified that backend endpoints were being deleted, but did not yet know how or why. Our on-call engineers were investigating two hypotheses.

We classified the event as critical, engaging incident response personnel to further investigate, coordinate resources, and drive customer workstreams.

We identified the automated cleanup operation deleting resources. With the problem known, we began mitigation efforts, beginning with stopping the automation that was executing the cleanup.

The cleanup job stopped on its own and did not delete everything. However, we continued to work to ensure further deletion was prevented.

We added resource locks and stopped delete requests at both the ARM and Automation levels, to prevent further deletion.

Initial recovery efforts began to recreate deleted resources. Note regarding recovery time - AOAI models are very large and can only be transferred via secure channels. Within a model pool (there are separate model pools per region per model) each deployment is serial (so the model copying time is a significant factor). Model pools themselves are done in parallel.

First communication posted to the Azure Status page. Communications were delayed due to initial difficulties scoping affected customers, and impact analysis gaps in our communications tooling.

Customer impact scope determined, and first targeted communications sent to customers via Service Health in the Azure portal.

GPT-3.5-Turbo started recovering in North Central US.

GPT-4o recovered in East US 2.

Majority of regions and models began recovering.

GPT-4 in majority of regions recovered and serving traffic.

Majority of models and regions recovered, error rates dropped to normal levels. GPT-4 recovered in all regions except Canada Central, Sweden Central, North Central US, UK South, Central US, and Australia East.

GPT-4 in North Central recovered.

Recovered in all regions except Sweden Central.

DALL-E restored in all regions.

GPT-4 in UK South is recovered.

GPT-4o recovered in Sweden Central.

All base models recovered, and service restoration completed across all affected regions.

Fine tuning model recovery across various regions.

All fine-tuning model deployments restored across affected regions.