We are experiencing an issue with Cloud Monitoring beginning at Monday, 2021-06-21 00:00 US/Pacific.

Incident Start: 21 June 2021 12:01 Incident End: 21 June 2021 21:55 Duration: 9 hours, 54 minutes Affected Services and Features: Google Cloud Monitoring, Cloud EKM Regions/Zones: Global Description: Google Cloud Monitoring experienced intermittent API errors with a gradually increasing error rate on a limited set of internal endpoints for 9 hours and 54 minutes. The Cloud Monitoring section in the Cloud Console experienced intermittent errors when loading pages due to the underlying API errors. This included the following pages; Monitoring Homepage, Dashboard Builder, GKE Dashboard, Metrics Explorer, Network Topology, Uptime Checks Service. Additionally, attempting to create the first Cloud EKM key [1] in a project would fail during this period. If a project already has (or at some point had) Cloud EKM keys, they can continue to create keys in those projects and use them. The root cause of the issue is suspected to be the rollout of a permission change which incorrectly restricted access to the underlying database which stores metadata information about the affected APIs. Customer Impact: Cloud Console errors on up to 10% of the listed page loads or equivalent API’s. Additional details: Workaround: Retrying failed requests with exponential backoff returned successful results for some customers. Workaround: Before attempting to create the first Cloud EKM key, running gcloud beta services identity create --service=cloudkms.googleapis.com --project $KEY_PROJECT_ID” prevented the failure. Reference(s): [1] https://cloud.google.com/service-usage/docs/reference/rest/v1beta1/services/generateServiceIdentity

The issue with Cloud Monitoring has been resolved for all affected projects as of Monday, 2021-06-21 21:46 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: We are experiencing an issue with Cloud Monitoring beginning at Monday, 2021-06-21 00:00 US/Pacific. Description: We are experiencing an issue with Cloud Monitoring beginning at Monday, 2021-06-21 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2021-06-21 23:00 US/Pacific with current details. Diagnosis: You should be able to see an API error about "method not available" in the Network Tab in your browser's developer console. Workaround: Trying to refresh or re-issue the query might help.

Summary: We are experiencing an issue with Cloud Monitoring beginning at Monday, 2021-06-21 00:00 US/Pacific. Description: We are experiencing an issue with Cloud Monitoring beginning at Monday, 2021-06-21 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2021-06-21 21:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: You should be able to see an API error about "method not available" in the Network Tab in your browser's developer console. Workaround: Trying to refresh or re-issue the query might help.