Google engineer are currently investigating a issue with the cloud networking product

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 19 June 2022 10:14 Incident End: 19 June 2022 13:39 Duration: 3 hours, 25 minutes Affected Services and Features: Google Cloud Networking. Regions/Zones: Global Description: Google Cloud Networking experienced an issue with the Cloud Armour config changes that got stalled for a duration of 3 hours, 25 minutes. From preliminary analysis, the root cause of the issue is Google Cloud Armor Configuration Server experienced errors in writing configs to the config storage system. Customer Impact: Customer configs propagation depending on the dos_cloud configs got delayed. Some impacted customers would not be able to make Cloud Armor config changes.

The issue with Google Cloud Networking has been resolved for all affected projects as of Sunday, 2022-06-19 14:27 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Google engineer are currently investigating a issue with the cloud networking product Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Sunday, 2022-06-19 14:30 US/Pacific. Diagnosis: Customer configs propagation depending on the dos_cloud configs is delayed. Customer will be able to make cloud armour config changes but expect higher latency for backlog to clear Workaround: n/a

Summary: Google engineer are currently investigating a issue with the cloud networking product Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Sunday, 2022-06-19 14:30 US/Pacific. Diagnosis: Customer configs propagation depending on the dos_cloud configs is delayed. Customer will be able to make cloud armour config changes but expect higher latency for backlog to clear Customer configs propagation depending on the dos_cloud configs is delayed. Workaround: n/a

Summary: Google engineer are currently investigating a issue with the cloud networking product Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Sunday, 2022-06-19 13:40 US/Pacific. Diagnosis: Customer are unable to make cloud armour config changes globally. Customer configs propagation depending on the dos_cloud configs is delayed. Workaround: n/a

Summary: Google engineer are currently investigating a issue with the cloud networking product Description: We are experiencing an issue with Google Cloud Networking beginning at Sunday, 2022-06-19 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Sunday, 2022-06-19 13:35 US/Pacific with current details. Diagnosis: Customer are unable to make cloud armour config changes globally. Customer configs propagation depending on the dos_cloud configs is delayed. Workaround: n/a

Summary: Google engineer are currently investigating a issue with the cloud networking product Description: We are experiencing an issue with Google Cloud Networking beginning at Sunday, 2022-06-19 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Sunday, 2022-06-19 13:35 US/Pacific with current details. Diagnosis: Customer are unable to make cloud armour config changes globally. Workaround: n/a

Summary: Google engineer are currently investigating a issue with the cloud networking product Description: We are experiencing an issue with Google Cloud Networking beginning at Sunday, 2022-06-19 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Sunday, 2022-06-19 13:35 US/Pacific with current details. Diagnosis: Customer are unable to make cloud amour config changes globally. Workaround: n/a