Google Cloud MAJOR

Support Systems Salesforce case updates fails with row lock error.

September 13, 2022 · 05:02 AM UTC – 01:22 AM UTC · Duration: 92h 20min

Affected Services

Google Cloud Support

Timeline

04:58 AM
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support (All Times US/Pacific) Incident Start: 12 September 2022 21:02 Incident End: 16 September 2022 17:22 Duration: 3 days, 20 hours, 20 minutes Affected Services and Features: Google Cloud Support systems Regions/Zones: Global Description: Google Cloud Support Systems experienced an intermittent issue where case updates from the portal were failing for a brief period. Retrying the same update may have succeeded in some cases. From preliminary analysis, the root cause of the issue is a large backlog of update actions that built up in the entitlement process. Customer Impact: Less than 0.1% of Google Cloud and Google Workspace customers experienced delays in contacting, providing and getting updates to and from Google Support Agents. The issue impacted support’s ability to send updates to customers via the case management system. Following are the scenarios in which errors were received. While sending emails to customers, the send action is unavailable due to errors. Error while changing case ownership. Error while posting comments on the case. Some customers experienced an error “There was an error adding your comment. Please try again” while updating cases from the support portal.
01:29 AM
The issue with Google Cloud Support has been resolved for all affected users as of Friday, 2022-09-16 17:27 US/Pacific. We thank you for your patience while we worked on resolving the issue.
11:49 PM
Summary: Support Systems Salesforce case updates fails with row lock error. Description: Our engineering team continues to investigate the issue with Cloud support systems and are currently working the mitigation. Customer impact: Customers are unable to update the support cases from the support portal. We will provide an update by Friday, 2022-09-16 18:15 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Support case updates are failing with error. Workaround: Retrying the update action successfully update the support case.