07:54 AM
Initial electrical failure in an onsite transformer, although various UPS batteries carried the load for several minutes.
07:58 AM
Customers began experiencing unavailability and delayed monitoring/log data.
08:07 AM
Datacenter and engineering teams engaged and initiated coordinated investigation across power, network, storage, and compute recovery workstreams.
09:31 AM
90% of facility power restored on generators.
11:26 AM
93% of facility power restored on generators.
11:29 AM
100% of facility power restored on generators with continued recovery of hosted services.
12:15 PM
Storage recovery started and dependent services started seeing recovery.
03:00 PM
Targeted remediation actions continued for remaining unhealthy services.
03:25 PM
Applied configuration changes to optimize processing of recovery events, to assist in expediting the recovery of compute services.
07:05 PM
Storage recovery was complete.
07:24 PM
Compute recovery system stabilized and fully operational, continued gradually restoring remaining nodes.
11:30 PM
Long tail of Compute nodes impact mitigated.
04:24 AM
All affected services confirmed mitigated.
03:42 AM
Power was transitioned back to utility power, restoring the datacenter to normal operations.