11:54 PM
Customer impact began, triggered by the configuration change.
12:08 AM
The issue was detected via our automated monitoring.
12:12 AM
Investigation commenced by our Azure Storage and Networking engineering teams.
02:33 AM
We performed steps to revert the configuration change for the Switzerland North region.
03:40 AM
The certificate was successfully reverted for the region.
04:00 AM
Validation of recovery was confirmed for the majority of impacted services, but a subset of resources were still unhealthy due to residual impact. We performed an extended health evaluation to confirm which remaining resources required recovery, and investigated the steps were needed to recover residual impact.
06:19 AM
We took additional time to recover remaining residual impact which included developing and validating new scripts to start resources that did not recover automatically.
11:24 AM
Performed necessary checks to test safe recovery actions for different types of resources.
02:15 PM
Recovery operations were performed for residual impact. This included executing mitigation scripts, and performing steps to safely reboot and recover remaining resources.
04:15 PM
Partial recovery for residual impact. Additional operations to recover the remaining impact included properly restarting the resources in a stopped state.
09:59 PM
Residual impact recovery activities and validation completed, confirming full recovery of all impacted services and customers.