Google Cloud Infrastructure is failing to push cloud configs

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 08 May 2022 23:24 Incident End: 09 May 2022 02:42 Duration: 3 hours, 18 minutes Affected Services and Features: Google Cloud Networking Google App Engine Google Cloud Composer Regions/Zones: Global Description: Multiple Google Cloud services experienced issues from 8 May 2022 23:24 to 9 May 2022 02:42 US/Pacific. From preliminary analysis, the root cause of the issue was network issues that caused tests to fail which led the configuration management system to treat the config as a bad config and prevented propagating configuration changes to Google Cloud infrastructure backend servers. Customer Impact: Google Cloud Networking - Customers modifying backend configuration for load balanced products or health check configuration may have seen their configuration accepted but not propagate, or fail after a timeout Google App Engine - Customers may have observed the error, DEADLINE_EXCEEDED - “Timed out waiting for the app infrastructure to become healthy” when attempting to deploy updates. Google Cloud Composer - Customers would have been unable to create Composer 1 and some Composer 2 "Private IP" environments (only those using VPC peering, environments using Private Service Connect were unaffected).

The issue with Google App Engine, Google Cloud Composer, Google Cloud DNS, Google Cloud Networking, Google Compute Engine, Service Directory has been resolved for all affected users as of Monday, 2022-05-09 02:42 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Google Cloud Infrastructure is failing to push cloud configs Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Monday, 2022-05-09 04:00 US/Pacific. Diagnosis: Customers may experience the following: 1) External HTTP/S Load Balancing, Cloud CDN - config changes will be accepted but fail to propagate 2) Cloud Armor rules may also be affected 3) App Engine Flex customers may see apps fail to deploy 4) Traffic Director may see configs fail to propagate 5) VM instance group health checks & functionality like autoscaling/autohealing may be affected. Product Impact: Google App Engine Flex: Impact/Diagnosis: Users may observe error, DEADLINE_EXCEEDED. “Timed out waiting for the app infrastructure to become healthy”. This occurs when attempting to deploy major version updates. Workaround: None. Google Cloud Networking Impact/Diagnosis: Cloud customers modifying backend configuration for load balanced products, or health check configuration, may see their configuration accepted but not propagate, or fail after a timeout. Workaround: None Cloud Composer: Impact: Users are unable to create Composer 1 and Composer 2 "Private IP" environments. Workaround: No workaround is available for Composer 1. For Composer 2 “Private IP” the workaround is to create environments with Private Service Connect. Composer 2 “Public IP” environment creations should be successful. Workaround: None

Summary: GICS is failing to push cloud configs Description: We are experiencing an issue with Google Cloud DNS, Service Directory beginning at Sunday, 2022-05-08 23:24 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2022-05-09 03:00 US/Pacific with current details. Diagnosis: Customer's might experience following: External HTTP/S Load Balancing, Cloud CDN - config changes will be accepted & fail to propagate Cloud Armor rules may also be affected Appengine flex customers may see apps fail to deploy Traffic Director may see configs fail to propagate VM instance group healthchecks & functionality like autoscaling/autohealing may be affected Workaround: None