Global: Cloud Armor Security Policy Change Propagation Failures

Mini Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 06 September 2022 07:30 Incident End: 06 September 2022 09:13 Duration: 1 hour, 43 minutes Affected Services and Features: Cloud Armor - Security Policy Regions/Zones: Global Description: Cloud Armor experienced security policy update failures globally for 1 hour, 43 minutes. From preliminary analysis, this was due to failures in a backend storage component which caused stuck operations for several downstream operations. Customer Impact: All changes made to security policies made by customers after 07:30 US/Pacific were not propagated. Affected customers would see their submitted change did not take effect until 09:30 US/Pacific. This was also visible in the log files. Existing security policies were unaffected.

The issue with Cloud Armor has been resolved for all affected users as of Tuesday, 2022-09-06 09:11 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Global: Cloud Armor Security Policy Change Propagation Failures Description: We are experiencing an issue with Cloud Armor beginning at Tuesday, 2022-09-06 07:48 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2022-09-06 09:25 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Changes to security policies performed by Customers are not propagating. Existing security policies are unaffected. Workaround: None at this time.