Multiple Cloud products are experiencing elevated error rates in us-east5 due to authentication failures

Mini Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 26 June 2023 at 10:15 Incident End: 26 June 2023 at 11:04 Duration: 49 minutes Affected Services and Features: CloudSQL Cloud Pub/Sub Pub/Sub Lite Cloud Bigtable Data Catalog Google Compute Engine Cloud Spanner Google Cloud Storage Regions/Zones: us-east5 Description: Google Cloud Identity and Access Management experienced elevated error rates that resulted in authentication failures for multiple Google Cloud services in the us-east5 region for a duration of 49 minutes. The trigger for this outage was a new deployment for some Cloud IAM workloads in the us-east5 region. The new workloads were incorrectly configured to use authentication keys whose corresponding verification keys were not yet distributed. As Cloud services detected the new IAM workloads and started to send them authentication requests, they received in return authentication proof tokens that could not be verified, resulting in failed workflows. The issue was mitigated by redirecting traffic from the affected IAM workloads. Customer Impact: Cloud SQL 1 failed update operation resulting in downtime for HA instance 4 failed creations Missing metrics for up to 50 minutes by <150 instances Cloud Pub/Sub Affected customers experienced elevated authentication errors. Pub/Sub Lite Affected customers experienced elevated authentication errors. Cloud Bigtable Affected customers experienced elevated 401 authentication errors. Data Catalog Affected customers experienced elevated 401 authentication errors. Google Compute Engine Affected customers experienced elevated 401 authentication errors. About 50% of users were impacted in us-east5-c (300 QPS of 600) from 10:21 -11:05 a.m. Cloud Spanner Affected customers experienced elevated 401 authentication errors. Google Cloud Storage -Affected customers experienced elevated 400 authentication errors. <1% of customers were affected from 10:20 - 11:09 a.m. <1% QPS of total traffic received errors during this time peaking at around 1100 QPS before the incident was mitigated.

The issue with Data Catalog, Google Cloud Bigtable, Google Cloud Pub/Sub, Google Cloud SQL, Cloud Spanner has been resolved for all affected users as of Monday, 2023-06-26 11:06 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Multiple Cloud products are experiencing issues in us-east5 Description: We are experiencing an issue with Google Cloud SQL, Google Cloud Bigtable, Data Catalog, Google Cloud Pub/Sub. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2023-06-26 12:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: None at this time. Workaround: None at this time.