Global: Delays creating or modifying load balancer configurations

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 31 March 2022 11:07 Incident End: 31 March 2022 15:13 Duration: 4 hours, 6 minutes, Affected Services and Features: Global External HTTP/S load balancing Cloud CDN Cloud Armor Appengine flex Regions/Zones: Global Locale Description: Cloud Load Balancers experienced delays creating or modifying configurations for 4 hours 6 minutes. From the preliminary analysis, the root cause of the issue is a rollout on the configuration management system that manages and pushes configuration for multiple Google infrastructure systems. The roll out caused an increase in the task counts which overloaded the configuration management system as all tasks pulled the configuration at the same time. Customer Impact: Affected customers would have experienced delays in creating or modifying the configuration of Global External HTTP(S) load balancers, Cloud Armor and Cloud CDN. App Engine Flexible apps might have failed to deploy.

The issue with Cloud Armor, Google App Engine, Google Cloud Networking has been resolved for all affected users as of Thursday, 2022-03-31 14:55 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Global: Delays creating or modifying load balancer configurations Description: Mitigation efforts are reducing the backlog of delayed configuration changes, which are slowly being processed. We will provide more information by Thursday, 2022-03-31 15:15 US/Pacific. Diagnosis: Affected customers will see delays in creating or modifying the configuration of Global External HTTP(S) load balancers, Cloud Armor and Cloud CDN. App Engine Flexible apps may fail to deploy. Configuration changes will be accepted, but not take effect until the issue is resolved. Existing configurations and deployments are not impacted. Workaround: Configuration changes will eventually take effect. Failed App Engine Flexible deployments can be retried.

Summary: Global: Delays creating or modifying load balancer configurations Description: Our engineering team is exploring multiple mitigations options currently. We will provide more information by Thursday, 2022-03-31 15:00 US/Pacific. Diagnosis: Affected customers will see delays in creating or modifying the configuration of Global External HTTP(S) load balancers, Cloud Armor and Cloud CDN. App Engine Flexible apps may fail to deploy. Configuration changes will be accepted, but not take effect until the issue is resolved. Existing configurations and deployments are not impacted. Workaround: Configuration changes will eventually take effect. Failed App Engine Flexible deployments can be retried.

Summary: Global: Unable to create or modify some Cloud Networking configurations Description: Mitigation work is still underway by our engineering team. Current data indicates that approximately 15% of projects globally are affected by this issue. We will provide more information by Thursday, 2022-03-31 14:00 US/Pacific. Diagnosis: Affected customers are not able to create or change External HTTP(S) load balancers. Cloud Armor and Cloud CDN have similar symptoms. App Engine Flexible apps may fail to deploy. In general, config change will be accepted, but will not take effect. Existing configurations are not impacted. Workaround: None at this time.

Summary: Global: Unable to create or modify some Cloud Networking configurations Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2022-03-31 13:00 US/Pacific. Diagnosis: Affected customers are not able to create or change External HTTP(S) load balancers. Cloud Armor and Cloud CDN have similar symptoms. App Engine Flexible apps may fail to deploy. In general, config change will be accepted, but will not take effect. Existing configurations are not impacted. Workaround: None at this time.

Summary: Global: Unable to create or modify regional external HTTP(S) load balancers Description: We are experiencing an issue with Google Cloud Networking beginning at Thursday, 2022-03-31 11:07 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-03-31 12:45 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers are not able to create or change Regional external HTTP(S) load balancer balancers. Existing balancers are not impacted. Workaround: None at this time.