Global: Delays provisioning SSL certificates for HTTPS load balancers

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 11 February 2022 12:00 Incident End: 11 February 2022 18:26 Duration: 6 hours, 26 minutes Affected Services and Features: Google Cloud Networking - Global External HTTPS load balancer SSL certificates Regions/Zones: Global Description: Google Cloud Networking experienced issues with global external HTTPS load balancer SSL certificate configuration updates for a duration of 6 hours, 26 minutes. This was triggered by a component rollback, which tripped a safety check that prevented changes from taking effect. The backlog required further intervention from operators to clear; when it did the issue was mitigated. Customer Impact: Customers experienced delays with initial provisioning of managed SSL certificates. Customers experienced issues with programming of newly uploaded, updated, or deleted SSL certificates that were not applied to the dataplane. Additional details: The issue was fully resolved on 11 February 2022 at 18:26 US/Pacific after: Modifying the safety check. Supervising the pipeline until it successfully completed, removing the delta and resolving the issue.

The issue with Cloud Networking has been resolved for all affected users as of Friday, 2022-02-11 18:26 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Global: Delays provisioning SSL certificates for HTTPS load balancers Description: Mitigation work is still underway by our engineering team. A configuration change is rolling out as a potential fix. We will provide more information by Friday, 2022-02-11 20:30 US/Pacific. Diagnosis: Affected customers may experience delays when provisioning new or updated certificates. Workaround: None at this time.

Summary: Global: Delays provisioning SSL certificates for HTTPS load balancers Description: We are experiencing an intermittent issue with Cloud Networking beginning at Friday, 2022-02-11 12:00 US/Pacific. Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We apologize to all who are affected by the disruption We will provide more information by Friday, 2022-02-11 18:00 US/Pacific. Diagnosis: Affected customers may experience delays when provisioning new or updated certificates. Workaround: None at this time.