Elevated latencies for network change propagation

We are investigating elevated latencies for network change propagation in the EU-WEST-1 Region. This is resulting in impact to other services, such as timeouts when pulling images. We will provide you with further information shortly.

[[The following AWS services have been affected by this event:]] Service: apigateway Status: Informational Service: athena Status: Informational Service: awswaf Status: Informational Service: cognito Status: Informational Service: deadline Status: Informational Service: dynamodb Status: Informational Service: ecr Status: Informational Service: ecs Status: Informational Service: eks Status: Informational Service: elasticache Status: Informational Service: elasticsearch Status: Informational Service: elb Status: Informational Service: elemental Status: Informational Service: emr Status: Informational Service: emrserverless Status: Informational Service: fargate Status: Informational Service: globalaccelerator Status: Informational Service: natgateway Status: Informational Service: personalize Status: Informational Service: rds Status: Informational Service: s3 Status: Informational Service: sagemaker Status: Informational

We can confirm elevated latencies for network change propagation in the EU-WEST-1 Region. This causes delays when attempting to connect to newly launched instances using their assigned public IPs, as well as delays in establishing new VPC Peering connections. Other AWS Services are also impacted by this issue and may be impacted by delayed or stuck workflows such as pulling container images. Engineers were automatically engaged and we are actively working on mitigating this issue. We plan to provide an update within the next 60 minutes.

We are seeing early signs of progressive recovery. We continue to work toward full recovery and will continue to provide updates as we work toward full mitigation. We recommend customers continue to retry failed requests. As recovery progresses, additional requests will succeed.

We continue to work toward fully mitigating the issue that is resulting in elevated latencies for network change propagation delays in the EU-WEST-1 Region. We have a high degree of confidence that our current mitigation efforts will fully mitigate the issue. Our current estimation is that these mitigations will complete within the next 60 to 120 minutes. We recommend customers continue to retry failed requests. We will provide additional information as recovery continues to progress.

We are seeing significant signs of recovery. We continue to work toward full resolution.

Between 7:08 AM and 10:48 AM PST, we experienced elevated latencies for network change propagation in the EU-WEST-1 Region. This was due to a delay in propagation of configuration updates on a sub-system which supports external and cross account connectivity. Customers experienced delays when attempting to connect to newly launched instances using their assigned public IPs, as well as delays in establishing new VPC Peering connections. Other AWS Services were also impacted by this issue and were impacted by delayed or stuck workflows such as pulling container images. Engineers were automatically engaged at 6:29 AM, prior to customer impact beginning. At 7:20 AM, root cause was identified. By 9:20 AM we began observing early signs of recovery. Our mitigation efforts completed at 10:43 AM and full recovery was observed at 10:48 AM. We recommend customers retry any failed requests. The issue has been resolved and all services are operating normally.

[[The following AWS services have been affected by this event:]] Service: apigateway Status: Operating normally Service: athena Status: Operating normally Service: awswaf Status: Operating normally Service: cognito Status: Operating normally Service: deadline Status: Operating normally Service: dynamodb Status: Operating normally Service: ecr Status: Operating normally Service: ecs Status: Operating normally Service: eks Status: Operating normally Service: elasticache Status: Operating normally Service: elasticsearch Status: Operating normally Service: elb Status: Operating normally Service: elemental Status: Operating normally Service: emr Status: Operating normally Service: emrserverless Status: Operating normally Service: fargate Status: Operating normally Service: globalaccelerator Status: Operating normally Service: natgateway Status: Operating normally Service: personalize Status: Operating normally Service: rds Status: Operating normally Service: s3 Status: Operating normally Service: sagemaker Status: Operating normally