Google engineering are investigating issues with Cloud Networking, starting at 2022-02-10 00:32:56 PST

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 10 February 2022 00:32 Incident End: 10 February 2022 06:34 Duration: 6 hours, 32 minutes Affected Services and Features: Google Cloud Networking, Google App Engine, Google Cloud Storage, Google Compute Engine Regions/Zones: asia-south1 Description: Google Cloud Networking experienced elevated latency in the asia-south1 region for 6 hours and 32 minutes. From preliminary analysis, the root cause of the issue was increased memory utilization related to a query workload. Customer Impact: Customers may have experienced elevated latency in the asia-south1 region between Google Cloud Load Balancers (GCLB) and downstream services including Google App Engine, Google Cloud Storage, and Google Compute Engine.

The issue with Google App Engine, Google Cloud Storage, Cloud Networking, Google Compute Engine has been resolved for all affected users as of Thursday, 2022-02-10 06:18 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Google engineering are investigating issues with Cloud Networking, starting at 2022-02-10 00:32:56 PST Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2022-02-10 06:30 US/Pacific. Diagnosis: Customer may see - increased latency. Workaround: None at this time.

Summary: Google engineering are investigating issues with Cloud Networking, starting at 2022-02-10 00:32:56 PST Description: We are experiencing an issue with Cloud Networking Thursday, 2022-02-10 00:32 US/Pacific Our engineering team is narrowing down the root cause and continues to investigate. Customer may see various error in the APAC region. We will provide an update by Thursday, 2022-02-10 04:00 US/Pacific with current details. Diagnosis: Customer may see - Time outs, 502 error, and slow response times, increased latency. Workaround: None at this time.

Summary: Google engineering are investigating issues with Cloud Networking, starting at 2022-02-10 00:32:56 PST Description: We are experiencing an issue with Cloud Networking Thursday, 2022-02-10 00:32 US/Pacific Our engineering team is narrowing down the root cause and continues to investigate. Customer may see various error in the APAC region. We will provide an update by Thursday, 2022-02-10 03:30 US/Pacific with current details. Diagnosis: Customer may see - Time outs, 502 error, and slow response times, increased latency. Workaround: None at this time.

Summary: Google engineering are investigating issues with Cloud Networking, starting at 2022-02-10 00:32:56 PST Description: We are experiencing an issue with Cloud Networking Thursday, 2022-02-10 00:32 US/Pacific Our engineering team continues to investigate the issue. Customer may see various error in the APAC region We will provide an update by Thursday, 2022-02-10 03:30 US/Pacific with current details. Diagnosis: Customer may see - Time outs, 502 error, and slow response times Workaround: None at this time.

Summary: Google engineering are investigating issues with Cloud Networking, starting at 2022-02-10 00:32:56 PST Description: We are experiencing an issue with Cloud Networking Thursday, 2022-02-10 00:32 US/Pacific Our engineering team continues to investigate the issue. Customer may see various error in the APAC region We will provide an update by Thursday, 2022-02-10 03:00 US/Pacific with current details. Diagnosis: Customer may see - Time outs, 502 error, and slow response times Workaround: None at this time.

Summary: Google engineering are investigating issues with Cloud Networking, starting at 2022-02-10 00:32:56 PST Description: We are experiencing an issue with Cloud Networking Thursday, 2022-02-10 00:32 US/Pacific Our engineering team continues to investigate the issue. Customer maybe see various error in the APAC region We will provide an update by Thursday, 2022-02-10 03:00 US/Pacific with current details. Diagnosis: Customer may see - Time outs, 502 error, and slow response times Workaround: None at this time.

Summary: Google engineering are investigating issues with Cloud Networking, Description: We are experiencing an issue with Cloud Networking Thursday, 2022-02-10 00:00 US/Pacific Our engineering team continues to investigate the issue. Customer maybe see various error in the APAC region We will provide an update by Thursday, 2022-02-10 02:30 US/Pacific with current details. Diagnosis: Customer may see - Time outs, 502 error, and slow response times Workaround: None at this time.

Summary: Google engineering are investigating issues with Cloud Networking, Description: We are experiencing an issue with Cloud Networking Thursday, 2022-02-10 00:00 US/Pacific Our engineering team continues to investigate the issue. Customer maybe see various error in the APAC region We will provide an update by Thursday, 2022-02-10 02:30 US/Pacific with current details. Diagnosis: Customer may see issue with APP engine, GCS, GCLP - Time outs, 502 error, and slow response times Workaround: None at this time.

Summary: Google engineering are investigating issues with Cloud Networking, Description: We are experiencing an issue with Cloud Networking Thursday, 2022-02-10 00:00 US/Pacific Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-02-10 02:30 US/Pacific with current details. Diagnosis: Customer may see with APP engine, GCS, GCLP - Time out, 502 error, and slow response Workaround: None at this time.