Post Incident Review (PIR) – Network Connectivity – Toronto edge site issues

Customer impact began, as the redundant device began experiencing the hardware failure, while the paired primary device had already been removed from rotation.

Our automation attempted to bring the paired primary device back into rotation and, due to system load balancing after route convergence, the impact was marginally reduced.

First automated alert raised of potential issues, however the scope of impact was not evident from the alert or initial investigations.

Network engineers investigating the issue took steps to remove congestion from the network peer, which reduced impact.

Additional customer reports received, following delays in submitting support requests via the Azure Portal.

Additional engineering teams were engaged to investigate and mitigate.

We began steps to manually remove the redundant device from rotation.

We isolated the faulty line card from the device. Network traffic was shifted to healthy routes, connectivity was restored, and we monitored the service health to ensure customers had been mitigated.