Post Incident Review (PIR) – Network Connectivity – Availability issues in East US

Customer impact began, triggered by a fiber cut causing network congestion which led to customers experiencing packet drops or intermittent connectivity. Our monitoring systems identified the impact immediately, so our on-call engineers engaged to investigate.

Our fiber provider was notified of the fiber cut and prepared for dispatch.

Mitigation efforts began identifying the impacted datacenters and redirecting traffic to healthier routes.

All customers using the East US region were notified about connectivity issues, even if their services were not directly impacted.

Mitigation efforts were successfully completed. All devices affected by the fiber cut were isolated, all customer traffic was using healthy paths and not experiencing congestion.

Customer impact recommenced, due to a tooling failure during the capacity repair process of the initial fiber cut.

This impact was mitigated after isolating the capacity that was incorrectly added by the tooling failure as part of the recovery process. Customers and services would have experienced full mitigation.

The underlying fiber cut was fully restored. We continued to monitor our capacity during the recovery process.

Fiber restoration efforts were completed. The incident was confirmed as mitigated.