Change Propagation Delays

We are investigating DNS resolution failures for some specific CloudFront distributions. We are actively investigating and will provide additional information in the next 30-60 minutes.

We can confirm errors for DNS resolution for some CloudFront distributions. During this time, customers may receive an NXDOMAIN response. Additionally, customers may also experience delayed propagation for changes to CloudFront distributions. We have identified the root cause of the issue and are actively working on multiple paths to resolving the errors. We have verified that our initial mitigation effort on a portion of the affected subsystem was successful, and we are actively working toward performing that mitigation across the fleet. We recommend customers continue to retry any failed requests while we work toward mitigation. A few services that use CloudFront distributions for delivering content may also be affected at this time.

We are seeing early signs of recovery, and continue to work toward full recovery.

We can confirm significant recovery for the DNS resolution errors for CloudFront distributions. Customers are still experiencing delays propagating changes to CloudFront distributions. We continue to work toward full recovery and will provide additional information in the next 30-60 minutes.

We can confirm full recovery for DNS resolution errors for CloudFront Distributions. Due to some of the mitigation actions we have taken, customers will experience delays in propagating changes to CloudFront distributions, including the creation of new CloudFront distributions. We continue to work toward recovering change propagation delays and will provide an update in the next 30-60 minutes.

We continue to work toward mitigating the current issue, which is delays in propagating changes to CloudFront distributions. Cache invalidation is not affected by this issue. In parallel, we are actively investigating options to increase the velocity of our recovery efforts. In order to ensure that we do not cause additional impact, we are proceeding cautiously but safely. During this time, customers can continue making changes but these changes will not propagate until we’ve fully mitigated this issue. We expect full recovery is a couple hours away.

We continue to apply mitigation steps for delays in propagating changes to CloudFront distributions. We are actively engaged and working on two parallel paths to mitigate the issue. Customers changes to CloudFront distributions will not propagate until we’ve fully mitigated this issue. End-user requests for content delivery from our edge locations and invalidations are being served normally at this stage. We expect full recovery in 2-3 hours. We will provide another update within 60 minutes, or sooner if information becomes available.

We are on track to apply mitigations towards resolving the change propagation delays to CloudFront distributions. We are implementing the mitigations in a striped manner on the affected systems that consume the configurations on the edge pops. Customers changes to CloudFront distributions will not propagate until we’ve fully mitigated this issue. End-user requests for content delivery from our edge locations and invalidations are being served normally at this stage. We expect full recovery in approximately 2 hours. We will provide another update within 60 minutes, or sooner if information becomes available.

We continue to work toward mitigating the change propagation delays to CloudFront distributions. We have further confirmed that the delayed propagation impact is limited to certain types of changes, such as creating/deleting distributions, or updating the distributions for DNS or TLS certificate related changes. End-user requests for content delivery from our edge locations and invalidations are being served normally at this stage. Our recovery is taking longer than anticipated in the previous update and we now estimate another 2 hours for full recovery. We will provide another update within 60 minutes, or sooner if information becomes available.

We continue to make progress towards mitigating the change propagation delays to CloudFront distributions. We have completed a successful configuration updates to a set of edge pops and are now propagating to edge pops in other stripes. The delayed propagation impact is limited to certain types of changes, such as creating/deleting distributions, or updating the distributions for DNS or TLS certificate related changes. End-user requests for content delivery from our edge locations and invalidations continue to be served normally at this stage. We estimate approximately 60 minutes for full recovery.

Between 12:20 PM and 1:55 PM PST, we experienced elevated DNS resolution errors for CloudFront distributions served from a subset of edge locations globally. During this time, customers may have received NXDOMAIN responses. Engineers were automatically engaged, immediately began investigating in multiple parallel paths and mitigated errors by taking the affected fleet of DNS servers out of service. Additionally, between 12:20 PM and 8:08 PM PST, we experienced longer than usual propagation times for changes to CloudFront configurations. The propagation delay was limited to certain types of changes, such as creating/deleting distributions, or updating the distributions for DNS or TLS certificate related changes. Following mitigation of the DNS resolution errors at 1:55 PM, end-user requests for content delivery from our edge locations were not affected and are being served normally. Invalidations operated normally throughout the event. Both issues have been resolved and all services are operating normally. We are confident this issue will not reoccur.