Virtual Private Cloud public IP packet loss affecting GCE instances in multiple regions

Mini Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support . (All Times US/Pacific) Impact Window #1: Region: asia-southeast1 Incident Start: 6 June 2023 at 07:35 Incident End: 6 June 2023 at 07:58 Duration: 23 minutes Impact Window #2: Region: asia-east1 Incident Start: 6 June 2023 at 08:02 Incident End: 6 June 2023 at 08:18 Duration: 16 minutes Impact Window #3: Region: europe-west1 Incident Start: 6 June 2023 at 08:08 Incident End: 6 June 2023 at 08:35 Duration: 27 minutes Affected Services and Features: Google Cloud Networking - Cloud VPN, Cloud NAT, Virtual Private Cloud (VPC) Regions/Zones: asia-southeast1, asia-east1, europe-west1 Description: Google Cloud Networking experienced elevated packet loss and latency which impacted multiple Cloud Networking services in asia-southeast1, asia-east1, europe-west1 for a maximum duration of 27 minutes. From preliminary analysis, the root cause was a configuration change for one of the load balancers, which directed traffic for VM public IPs to an incorrect data plane component. Customer Impact: Google Cloud Networking VM to VM public IP and VM to External TCP/UDP Network Load Balancer traffic experienced up to ~50% packet loss across above three regions spanning the same durations. Cloud VPN Affected customers experienced up to ~55% packet loss across above three regions spanning the same durations. Cloud NAT Affected customers experienced up to ~60% packet loss across above three regions spanning the same durations.

The issue with Virtual Private Cloud (VPC) has been resolved for all affected projects as of Tuesday, 2023-06-06 08:47 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Virtual Private Cloud public IP packet loss affecting GCE instances in multiple regions Description: We are experiencing an issue with Virtual Private Cloud (VPC) beginning at Tuesday, 2023-06-06 07:35 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2023-06-06 10:06 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Packet loss when connecting to GCE instances or Network Load Balancers using public IP addresses Workaround: None at this time.