High latency and increased error rates in northamerica multi-region.

Incident Start: 22 June 2021 19:38 Incident End: 22 June 2021 20:51 Duration: 1 hours, 13 minutes Affected Services and Features: Google Cloud Firestore Regions/Zones: nam5 multi-region Description: Google Cloud Firestore experienced elevated latency and error rates on up top 50% of projects in the nam5 multi-region for a duration of 1 hour and 13 minutes. The root cause of the issue was a traffic spike from an internal daily batch job which performs quota adjustments. This caused lock contention on the backend databases which, in conjunction with poor traffic distribution and misdirected traffic, ultimately overloaded the service. Customer Impact: Affected customers may see “Deadline Exceeded” errors and degraded performance.

The issue with Cloud Firestore has been resolved for all affected users as of Tuesday, 2021-06-22 21:53 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: High latency and increased error rates in northamerica multi-region. Description: We are experiencing an intermittent issue with Cloud Firestore beginning at Tuesday, 2021-06-22 19:38 US/Pacific. At the moment we see performance improvements and error rate is back to its original state. Our engineering team continues to monitor the situation. We will provide an update by Tuesday, 2021-06-22 22:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers might see 'Deadline Exceeded' errors and degraded performance. Workaround: None at this time.

Summary: High latency and increased error rates in northamerica multi-region. Description: We are experiencing an intermittent issue with Cloud Firestore beginning at Tuesday, 2021-06-22 19:38 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2021-06-22 21:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers might see 'Deadline Exceeded' errors and degraded performance. Workaround: None at this time.