Spectrum Access System Issues

Mini Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. (All Times US/Pacific) Impact Period 1: Incident Start: 3 March 2023 10:40 Incident End: 3 March 2023 12:30 Impact Period 2: Incident Start: 3 March 2023 17:00 Incident End: 3 March 2023 18:00 Cumulative Duration: 2 hours, 50 minutes Affected Services and Features: Networking - Spectrum Access System Regions/Zones: Global Description: Google Spectrum Access System (SAS) experienced elevated heartbeat latency and error rates for a duration of 2 hours, 50 minutes globally. From preliminary analysis, the root cause of the issue was caused by transaction failures from an upstream internal database. Our engineering team is reviewing migrating to an alternative system to prevent a future recurrence. Customer Impact: Affected customer’s Citizens Broadband Radio Service Device (CBSD) experienced elevated latency and error rates from the Google SAS, including timeouts and RESEND errors in the heartbeat response. If a CBSD fails to receive a successful heartbeat response before their transmitExpireTime, they lose their grant. In the event of a CBSD grant loss, additional time might have been required in order for the CBSD to successfully obtain a new grant. Additional details: Post-outage, if a CBSD lost their grant on 03 March 2023, it can relinquish and request a new one.

The issue with Spectrum Access System has been resolved for all affected users as of Friday, 2023-03-03 12:20 US/Pacific. Customers should not be experiencing failing Heartbeat Requests after Friday, 2023-03-03 12:20:00 US/Pacific. Customer CBSDs experienced elevated latency and error rates from the Google SAS, including timeouts and RESEND errors in the heartbeat response. There was no workaround during the outage. Post-outage, if a CBSD loses their grant, they can relinquish it and re-request it with the same parameters. Then the SAS should grant it to them immediately. We thank you for your patience while we worked on resolving the issue.

Summary: Spectrum Access System Issues Description: We've received a report of an issue with Spectrum Access System as of Friday, 2023-03-03 11:56 US/Pacific. We will provide more information by Friday, 2023-03-03 13:30 US/Pacific. Diagnosis: None at this time. Workaround: None at this time.