Logs delayed in europe-west3

Mini Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 14 December 2022 03:19 Incident End: 14 December 2022 06:30 Duration: 3 hours, 11 minutes Affected Services and Features: Cloud Logging Eventarc BigQuery Google Cloud Storage Google Cloud PubSub Regions/Zones: europe-west3 Description: Cloud Logging experienced elevated latency with log ingestion in europe-west3 for a duration of 3 hours, 11 minutes. From preliminary analysis, the root cause of the issue was due to an invalid cross project sink which resulted in a backlog of subscriber messages. The issue was fully mitigated once invalid sink was deleted and the backlog cleared. Google will be completing a comprehensive incident report in the following days that will provide a full root cause. Customer Impact: Affected customers may have experienced delayed log entries in Log Explorer or other export destinations for the duration of the incident.

The issue with Cloud Logging, Eventarc, Google BigQuery has been resolved for all affected users as of Wednesday, 2022-12-14 06:28 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Logs delayed in europe-west3 Description: Mitigation work is still underway by our engineering team. The mitigation is expected to complete by Wednesday, 2022-12-14 08:00 US/Pacific. We will provide more information by Wednesday, 2022-12-14 08:00 US/Pacific. Diagnosis: Log entries written in europe-west3 are delayed by 1+ hours. They may not show up in Log Explorer or other export destinations. Workaround: None at this time.

Summary: Logs delayed in europe-west3 Description: We are experiencing an issue with Cloud Logging, Eventarc, Google BigQuery beginning at Wednesday, 2022-12-14 03:19 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2022-12-14 06:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Log entries written in europe-west3 are delayed by 1+ hours. They may not show up in Log Explorer or other export destinations. Workaround: None at this time.

Summary: Delay in ingesting the logs in europe-west3 Description: We are experiencing an issue with Cloud Logging and Google BigQuery. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2022-12-14 05:15 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers would experience a delay in logs visibility. Workaround: None at this time.