Google Cloud MAJOR
Global : Cloud Run is experiencing elevated latency and deployment failures
April 2, 2023 · 12:40 AM UTC – 03:25 AM UTC · Duration: 2h 45min
Affected Services
Cloud Run
Timeline
07:30 PM
Mini Incident Report
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.
(All Times US/Pacific)
Incident Start: 1 April 2023 16:40
Incident End: 1 April 2023 19:25
Duration: 2 hours, 45 minutes
Affected Services and Features:
Cloud Run - Deployments
Regions/Zones: Global
Description:
Cloud Run experienced elevated latency and deployment failures for a duration of 2 hours, 45 minutes globally. From preliminary analysis, the root cause of the issue was an unexpected increase of requests which caused higher than normal latency and errors.
Customer Impact:
Affected customers experienced elevated serving latency, 500 errors, and deployment failures.
Additional details:
As a workaround, some customers who experienced deployment failures could try redeploying their applications although multiple attempts might have been needed to successfully redeploy.
03:44 AM
The issue with Cloud Run has been resolved for all affected users as of Saturday, 2023-04-01 19:25 US/Pacific.
We thank you for your patience while we worked on resolving the issue.
02:25 AM
Summary: Global : Cloud Run is experiencing elevated latency and deployment failures
Description: Our Engineering team identified a fix and mitigation work is currently underway.
We do not have an ETA for mitigation at this point.
We will provide more information by Saturday, 2023-04-01 20:00 US/Pacific.
Diagnosis: Affected customers will experience elevated serving latency, 500 errors and deployment failures.
Workaround:
No workarounds at this time for customers experiencing serving latency.
Customers who are experiencing deployment failures can try redeploying their applications
02:11 AM
Summary: Global : Cloud Run is experiencing elevated latency and deployment failures
Description: We are experiencing an issue with Cloud Run beginning at Saturday, 2023-04-01 16:40 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Saturday, 2023-04-01 18:45 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Affected customers will experience elevated serving latency, 500 errors and deployment failures.
Workaround:
No workarounds at this time for customers experiencing serving latency.
Customers who are experiencing deployment failures can try redeploying their applications