Increased Error Rates

We are investigating increased error rates in the EU-CENTRAL-2 Region.

[[The following AWS services have been affected by this event:]] Service: airflow Status: Informational Service: appconfig Status: Informational Service: appsync Status: Informational Service: athena Status: Informational Service: awswaf Status: Informational Service: batch Status: Informational Service: clientvpn Status: Informational Service: cloudtrail Status: Informational Service: cloudwatch Status: Informational Service: codedeploy Status: Informational Service: datasync Status: Informational Service: ec2 Status: Informational Service: ecr Status: Informational Service: ecs Status: Informational Service: eks Status: Informational Service: elasticsearch Status: Informational Service: elb Status: Informational Service: emr Status: Informational Service: fargate Status: Informational Service: firehose Status: Informational Service: glacier Status: Informational Service: glue Status: Informational Service: kafka Status: Informational Service: kinesis Status: Informational Service: kms Status: Informational Service: lambda Status: Informational Service: mgn Status: Informational Service: privatelink Status: Informational Service: route53 Status: Informational Service: sagemaker Status: Informational Service: servicediscovery Status: Informational Service: storagegateway Status: Informational Service: transfer Status: Informational Service: transitgateway Status: Informational Service: vpclattice Status: Informational

We can confirm substantial error rates for PUT and GET requests to Amazon S3 in the EU-CENTRAL-2 Region. Engineers engaged immediately based on automated alarming. We have triangulated the issue to a subsystem responsible for assembling objects from bytes in storage. We have begun implementing mitigations, and are observing some improvement in error rates. We continue to work to identify the root cause, and are working on multiple parallel paths to fully mitigate the issue. Other AWS Services (such as EC2 launches) that rely on S3 are also affected by this issue. Existing EC2 instances are unaffected by this issue. We will provide another update by 12:45 PM PST, or sooner if we have additional information to share.

We are seeing early signs of recovery and continue to monitor and work toward full recovery.

Between 11:27 AM and 12:20 PM PST we experienced substantial error rates for S3 PUT/GET requests in EU-CENTRAL-2 Region. Engineers were engaged immediately based on automated alarming. We identified the root cause as an issue with a subsystem responsible for assembling objects bytes in storage. At 12:04 PM PST, we implemented mitigations and began observing early signs of recovery for S3. Error rates continued to improve, and other AWS Services continued to recover until 12:50 PM PST when we observed full recovery. We continue to work toward backfilling Cloudwatch logs, and expect that to continue over the next couple hours. We recommend customers retry any failed requests. The issue has been resolved and all services are operating normally.

[[The following AWS services have been affected by this event:]] Service: airflow Status: Operating normally Service: appconfig Status: Operating normally Service: appsync Status: Operating normally Service: athena Status: Operating normally Service: awswaf Status: Operating normally Service: batch Status: Operating normally Service: clientvpn Status: Operating normally Service: cloudtrail Status: Operating normally Service: cloudwatch Status: Operating normally Service: codedeploy Status: Operating normally Service: datasync Status: Operating normally Service: ec2 Status: Operating normally Service: ecr Status: Operating normally Service: ecs Status: Operating normally Service: eks Status: Operating normally Service: elasticsearch Status: Operating normally Service: elb Status: Operating normally Service: emr Status: Operating normally Service: fargate Status: Operating normally Service: firehose Status: Operating normally Service: glacier Status: Operating normally Service: glue Status: Operating normally Service: kafka Status: Operating normally Service: kinesis Status: Operating normally Service: kms Status: Operating normally Service: lambda Status: Operating normally Service: mgn Status: Operating normally Service: privatelink Status: Operating normally Service: route53 Status: Operating normally Service: sagemaker Status: Operating normally Service: servicediscovery Status: Operating normally Service: storagegateway Status: Operating normally Service: transfer Status: Operating normally Service: transitgateway Status: Operating normally Service: vpclattice Status: Operating normally