Chronicle Security, Chronicle SOAR. High Remote Procedure Calls (RPC) error rate in Multiregion/us

Mini Incident Report We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 28 February 2024 23:50 Incident End: 29 February 2024 09:50 Duration: 10 hours Affected Services and Features: Chronicle Security - Security Information and Event Management (SIEM) Chronicle Security Orchestration, Automation and Response (SOAR) Regions/Zones: US/multi-region Description: Chronicle Security SIEM and Chronicle SOAR experienced elevated Remote Procedure Calls (RPC) error rate for Backstory API calls in US/multi-region for a total duration of 10 hours. A new service was activated which caused additional traffic to be generated and resource contention, which impacted overall RPC performance, latency, and error rate. Our engineers identified a subset of the root cause processes and eliminated them on Thursday, 29 February 2024 06:18 US/Pacific. However, some of these processes were missed in the initial analysis, causing additional traffic and resource contention at 7:13. Engineers took additional steps, further mitigating the issue at 09:50. At this time, we do not believe any additional actions are needed to prevent recurrence of this issue. Customer Impact: Backstory API calls would have failed with Remote Procedure Calls (RPC) errors.

The issue with Chronicle Security, Chronicle SOAR has been resolved for all affected projects as of Thursday, 2024-02-29 08:20 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Chronicle Security, Chronicle SOAR. High Remote Procedure Calls (RPC) error rate in Multiregion/us Description: We are experiencing an issue with Chronicle Security, Chronicle SOAR. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2024-02-29 09:00 US/Pacific with current details. Diagnosis: Impacted users may experience high failure rate for Backstory API calls Workaround: None at this time.