Post Incident Review (PIR) – Entra Privileged Identity Management – Customers experiencing API failures

The configuration change deployment was initiated.

The configuration change was progressed to the ring with the heavy workload.

Initial customer impact began.

Automated alerts received for intermittent, low volume of errors prompting us to start investigating.

We attempted isolated restarts on impacted database instances in an effort to mitigate low-level impact.

Automated monitoring alerted us to elevated error rates and a full incident response was initiated.

We identified that calls to the database were intermittently timing out. Traffic volume appeared to be normal with no significant surge detected however we observed the spike in CPU utilization.

Mitigation efforts began, including beginning to scale out the impacted environments.

Scale out efforts were observed as decreasing error rates but not completely eliminating failures. Further instance restarts provided temporary relief.

Scaling efforts continued. We engaged our database engineering team to help investigate.

While we did not correlate the deployment to this incident, we initiated a rollback of the configuration change.

Scale-out efforts completed.

Service availability telemetry was showing improvements. Some customers began to report recovery.

Customer impact confirmed as mitigated, after rollback of configuration change had completed and error rates had returned to normal levels.