europe-west2-c: Cloud networking is currently experiencing issues.

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support (All Times US/Pacific) Incident Start: 03 September 2021 05:40 Incident End: 03 September 2021 06:20 Duration: 40 minutes Affected Services and Features: Cloud Networking Compute Engine Persistent Disk Kubernetes Engine App Engine Cloud Run Cloud Bigtable Cloud Storage Cloud Filestore Cloud MemoryStore Cloud Monitoring Cloud VPN Regions/Zones: europe-west2-c Description: Google Cloud Networking experienced elevated packet loss and latency in europe-west2-c for a duration of 40 minutes. Additional Cloud services that use Cloud Networking experienced similar symptoms. From preliminary analysis, the root cause of the issue was a sudden redirection of traffic from a datacenter that was experiencing an issue. Customer Impact: Affected customers may have experienced elevated packet loss and latency as their traffic was temporarily redirected. Google Compute Engine instances experienced failed connectivity and packet loss on ingress, egress, and inter-zonal traffic. Autoscaling was also impacted and may have failed to make decisions. Persistent Disk experienced elevated I/O latency on regional and zonal disks Google Kubernetes Engine clusters experienced 30% of zonal and 5% of regional clusters reporting as unhealthy. Google App Engine and Cloud Run served elevated 500 errors within europe-west2. Cloud Bigtable experienced an elevated error rate and latency on operations within europe-west2. Cloud SQL experienced issues connecting to instances in europe-west2. Affected instances may have lost telemetry data, and HA instances failed over if the primary was in the affected zone. Google Cloud Storage experienced elevated read errors of up to 2.3% in europe-west2. Cloud Filestore instances within europe-west2-c may have been unreachable. Cloud MemoryStore instances within europe-west2 may have been unreachable. Cloud Monitoring experienced delayed or missing metrics from resources in the zone. Cloud VPN experienced a short duration where tunnels went down for a short time before being reestablished.

The issue with Cloud Networking has been resolved for all affected users as of Friday, 2021-09-03 06:20 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: europe-west2-c: Cloud networking is currently experiencing issues. Description: On further investigation we have confirmed the issue is affecting a single zone europe-west2-c. Our engineering team is currently investigating the issue and confirming the current impact to our users. We will provide an update by Friday, 2021-09-03 07:30 US/Pacific with current details. Diagnosis: Customer may high latency and packet loss. Workaround: Other Zone and region can be utilized for workaround.

Summary: europe-west2: Cloud networking is currently experiencing issues. Description: We are experiencing an issue with Cloud Networking , beginning at Friday, 2021-09-03 0535 US/Pacific Our engineering team is currently investigating the issue and confirming the current impact to our users. We will provide an update by Friday, 2021-09-03 07:30 US/Pacific with current details. Diagnosis: Customer may high latency and packet loss. Workaround: Other Zone and region can be utilized for workaround.

Summary: Cloud networking issues being investigated impacting some user in eu-west2c Description: We are experiencing an issue with Cloud Networking , beginning at Friday, 2021-09-03 0535 US/Pacific Our engineering team continues to investigate the issue. We will provide an update by Friday, 2021-09-03 06:59 US/Pacific with current details. Diagnosis: Customer may high latency and packet loss Workaround: Other Zone and region are unaffected