Google Cloud MAJOR

us-central1, us-west1, us-west1, us-east1: Cloud Spanner missing stackdriver CPU, memory and disk metrics.

November 13, 2021 · 02:45 AM UTC – 04:39 AM UTC · Duration: 25h 54min

Affected Services

Cloud Spanner

Timeline

09:26 PM
We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 12 November 2021 18:45 Incident End: 13 November 2021 20:39 Duration: 1 day, 1 hours, 54 minutes Affected Services and Features: Cloud Spanner - Missing/Incorrect Cloud Monitoring metrics for CPU, memory and disk Regions/Zones: us-central1 us-east1 Global Description: Cloud Spanner experienced unavailable and incorrect monitoring metrics for instance resources including cpu, memory and disk for a duration of 1 day, 1 hours and 54 minutes. From preliminary analysis, the root cause of the issue is a rollout flag which incorrectly modified how metrics are exported. Customer Impact: us-central1: Experienced missing metrics during the impact duration until mitigation at 13 November 2021 18:30 us-east1: Experienced missing metrics during the impact duration until mitigation at 13 November 2021 17:45 All other regions: Many customers globally experienced metrics that incorrectly under-reported actual values until full resolution at 13 November 2021 20:39
05:17 AM
The issue with Cloud Spanner has been resolved for all affected users as of Saturday, 2021-11-13 21:16 US/Pacific. We thank you for your patience while we worked on resolving the issue.
12:47 AM
Summary: us-central1, us-west1, us-west1, us-east1: Cloud Spanner missing stackdriver CPU, memory and disk metrics. Description: Mitigation work is currently underway by our engineering team. Current data indicates that customers in us-central1, us-west1, us-west1 and us-east1 are affected by this issue. We will provide more information by Saturday, 2021-11-13 23:00 US/Pacific. Diagnosis: Missing Stackdriver metrics on the dashboard. Workaround: None at this time.