GKE API seeing high latency and errors in us-central1

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 14 June 2022 03:03 Incident End: 14 June 2022 06:22 Duration: 3 hours, 19 minutes Affected Services and Features: Google Kubernetes Engine, Cloud Data Fusion, Cloud Logging Regions/Zones: us-central1 Description: Google Kubernetes Engine (GKE) experienced increased latency and errors from the GKE API in us-central1 for 3 hours and 19 minutes. From preliminary analysis, the root cause of the issue was an unexpected increase in backend traffic that was compounded by clients retrying failed requests. The issue was mitigated by scaling up the GKE control plane to handle the increased traffic. Customer Impact: Google Kubernetes Engine (GKE) customers may have experienced increased latency and errors for GKE API calls via gcloud and the Google Cloud Console UI. GKE cluster workloads continued to run; however customers may have been unable to perform operations such as confirm cluster status, trigger upgrades, view workloads, or create/delete clusters and node pools. Cloud Data Fusion customers may have experienced instance creation and deletion failures in us-central1. Cloud Logging customers may have experienced increased latency for suggested searches for GKE clusters in us-central1.

The issue with Google Kubernetes Engine has been resolved for all affected users as of Tuesday, 2022-06-14 06:17 US/Pacific. If customers are still experiencing issues, please raise the case via normal channels . We thank you for your patience while we worked on resolving the issue.

Summary: GKE API seeing high latency and errors in us-central1 Description: Mitigation work is still underway by our engineering team. Customers who are able to use another region are advised to do so. We will provide more information by Tuesday, 2022-06-14 06:35 US/Pacific. Diagnosis: Users will see high errors and latency from the GKE API in us-central1. Retries may be successful. Workaround: None at this time

Summary: GKE API seeing high latency and errors in us-central1 Description: Mitigation work is still underway by our engineering team. Customers who are able to use another region are advised to do so. We will provide more information by Tuesday, 2022-06-14 06:10 US/Pacific. Diagnosis: Users will see high errors and latency from the GKE API in us-central1. Retries may be successful. Workaround: None at this time

Summary: GKE API seeing high latency and errors in us-central1 Description: Mitigation work is still underway by our engineering team. The mitigation is expected to complete by Tuesday, 2022-06-14 05:30 US/Pacific.} We will provide more information by Tuesday, 2022-06-14 05:31 US/Pacific. Diagnosis: Users will see high errors and latency from the GKE API in us-central1. Retries may be successful. Workaround: None at this time

Summary: GKE API seeing high latency and errors in us-central1 Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Tuesday, 2022-06-14 05:00 US/Pacific. Diagnosis: Users will see high errors and latency from the GKE API in us-central1. Retries may be successful. Workaround: None at this time

Summary: GKE API seeing high latency and errors in us-central1 Description: We are experiencing an issue with Google Kubernetes Engine beginning at Tuesday, 2022-06-14 03:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2022-06-14 04:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Users will see high errors and latency from the GKE API in us-central1. Retries may be successful. Workaround: None at this time