Intermittent missing or delayed EC2 instance and status check metrics

We are experiencing intermittent missing or delayed EC2 instance and status check metrics in the US-EAST-1 Region. Alarms on delayed or missing metrics may transition into an INSUFFICIENT_DATA state. We are taking multiple parallel paths to mitigate this issue. While underlying resources are not affected by this issue, customers with automated actions based off of delayed or missing metric data may see their automations start. EC2 APIs are not impacted and therefore EC2 AutoScaling will not be affected by this issue.

We can confirm issues with intermittent missing and/or delayed EC2 instance metrics and status checks in the US-EAST-1 Region. While existing instances are unaffected by this issue and operating normally, metrics and status checks may be delayed or reporting INSUFFICIENT_DATA. We have identified the issue to be in an underlying subsystem responsible for publishing EC2 metric data to CloudWatch. Engineers were automatically engaged, and continue to investigate multiple paths to mitigate the issue in parallel. We recommend customers treat the INSUFFICIENT_DATA state as missing data instead of an alarm breach, especially when configuring the alarm to stop, terminate, reboot, or recover an instance. More information is available <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UsingAlarmActions.html">here</a>. While we do not have a firm ETA for resolution, we will provide another update by 12:30 PM, or sooner if we have additional information to share.

We are seeing early signs of recovery and continue to work toward full resolution. We will continue to provide updates.

We can confirm significant signs of recovery, and continuing to monitor to ensure stability. At this time, missing/delayed metrics and instance status checks are recovered. We are actively working to backfill delayed data.

Between 7:00 AM and 12:05 PM PST, we experienced errors while publishing EC2 instance metrics and status checks in the US-EAST-1 Region. This issue resulted in metrics and status checks to be delayed or report INSUFFICIENT_DATA. EC2 APIs and instances were unaffected by this issue and continue to operate normally. We were automatically engaged at 7:05 AM and began identifying multiple parallel paths to mitigate the issue. By 7:20 AM, we identified that the issue was related to an underlying subsystem responsible for publishing EC2 metric data to CloudWatch. By 12:03 PM, we completed our mitigation efforts and observed full recovery at 12:05 PM. New metrics are being published as expected. Delayed metrics are in the process of backfilling and may take a few hours to fully complete. The issue has been resolved and the service is operating normally.