us multiregion: Elevated tail latency on read operations for GCS buckets

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 17 December 2021 09:10 Incident End: 17 December 2021 22:11 Incident Duration: 13 hours, 1 minute Affected Services and Features: Google Cloud Storage (GCS) - Read and Write object operations Regions/Zones: US multi-region Description: Google Cloud Storage buckets experienced elevated tail latency on read and write object operations for a total duration of 13 hours, 1 minute. From preliminary analysis, an internal job incorrectly placed a significant load on the backend database. This originally presented as increased tailed latency. The database was significantly overloaded leading to elevated 5xx errors for a 47 minute period between 10:50 and 11:37. The 5XX errors were resolved at 11:37 by adding additional resources and stopping the offending job. However, the job caused an extended backlog of work, which took until 22:11 to clear. Customer Impact: Elevated 5xx errors related to read timeouts between 10:50 and 11:37 Elevated tail latency on read operations from GCS buckets for the duration of the incident.

The issue with Google Cloud Storage has been resolved for all affected users as of Friday, 2021-12-17 11:36 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: us multiregion: Elevated tail latency on read operations for GCS buckets Description: We believe the issue with Google Cloud Storage is partially resolved and the error rates decreased significantly. We do not have an ETA for full resolution at this point. We will provide an update by Friday, 2021-12-17 13:30 US/Pacific with current details. Diagnosis: Affected customers may experience elevated tail latency on read operations for buckets in the us multiregion. Workaround: Retrying failed or slow requests may succeed due to the relatively low error rate.

Summary: us multiregion: Elevated tail latency on read operations for GCS buckets Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Friday, 2021-12-17 13:00 US/Pacific. Diagnosis: Affected customers may experience elevated tail latency on read operations for buckets in the us multiregion. Workaround: Retrying failed or slow requests may succeed due to the relatively low error rate.

Summary: us: Elevated latency in us GCS buckets Description: We are experiencing an issue with Google Cloud Storage beginning at Friday, 2021-12-17 09:00 US/Pacific. Our engineering team continues to investigate the issue. Current estimates indicates that 1% of GCS requests are impacted. We will provide an update by Friday, 2021-12-17 13:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers may experience elevated tail latency on their buckets in the us multiregion. Workaround: None at this time.

Summary: us: Elevated latency in us GCS buckets Description: We are investigating a potential issue with Google Cloud Storage. We will provide more information by Friday, 2021-12-17 11:56 US/Pacific. Diagnosis: Affected customers may experience elevated latency on their buckets in the us multiregion. Workaround: None at this time.