Connectivity Issues in Availability Zone (cac1-az1) in the CA-CENTRAL-1 Region.

We are investigating connectivity issues affecting instances in a single Availability Zone (cac1-az1) in the CA-CENTRAL-1 Region.

We can confirm connectivity issues affecting a subset of EC2 instances and EBS volumes in a single Availability Zone (cac1-az1) in the CA-CENTRAL-1 Region. Engineers were automatically engaged and immediately began investigating. We have identified the root cause as a power issue, and are actively working to restore power. Other AWS Services are also affected by this issue, and are experiencing connectivity issues as well as increased error rates and latencies for some control plane operations. If possible, we recommend using an alternate Availability Zone while we work to fully mitigate this issue. We will provide another update in the next 60 minutes, or sooner if we have additional information to share.

We have restored power and are seeing signs of recovery. We continue to work toward recovering all affected EC2 Instances and impaired EBS volumes and will provide another update as soon as we have additional information to share.

Starting at 2:31 AM PDT, we experienced power and network connectivity issues for instances and EBS volumes in a single Availability Zone (cac1-az1) in the CA-CENTRAL-1 Region due to a power issue. By 3:58 AM PDT, power and networking connectivity had been restored to the majority of affected instances, and degraded performance for the majority of affected EBS volumes had been resolved. A subset of remaining instances and volumes are hosted on hardware which are adversely affected by the loss of power and we continue to work to recover all affected remaining resources. For immediate recovery, we recommend replacing any remaining affected resources if possible. We will provide another update in the next 60 minutes, or sooner if we have additional information to share.

We continue to work to fully restore power to impacted resources in a single Availability Zone (cac1-az1) in the CA-CENTRAL-1 Region. We have recovered the majority of impaired resources and are now working to confirm stability before we call full recovery. For immediate recovery, we recommend replacing any remaining affected resources if possible. We will provide another update as soon as we have additional information to share.

Between 2:31 AM and 5:43 AM PDT, we experienced power and network connectivity issues for instances and resources in a single Availability Zone (cac1-az1) in the CA-CENTRAL-1 Region. Our engineering team was engaged and immediately began working on mitigation while simultaneously investigating the root cause of this impact. Other affected AWS services took immediate mitigation actions aimed at reducing increased error rate and latencies for control plane operations. By 3:58 AM PDT, power and networking connectivity had been restored to the majority of affected instances, and the majority of affected resources were recovered. For the remaining subset of impacted resources, we worked to carefully validate stability while recovering them. By 5:43 AM PDT we had fully recovered all impacted resources. This issue has been resolved and the service is operating normally.