Google Cloud MAJOR
Issue resolved: HTTP load balancing returning error for some users in Europe.
July 27, 2021 · 12:15 PM UTC – 05:05 AM UTC · Duration: 16h 50min
Affected Services
Google Cloud NetworkingCloud Load Balancing
Timeline
04:12 PM
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support
(All Times US/Pacific)
Incident Timeline:
27 June 2021 04:15 - 05:33
27 June 2021 06:56 - 07:41
Total Duration:
2 hours 3 minutes
Affected Services and Features:
Google Cloud Load Balancing
Regions/Zones:
Western Europe
Description:
Google Cloud Load Balancing (GCLB) experienced increased TLS handshake errors or connection terminations for HTTPS traffic served from load balancers in Western Europe for a duration of 2 hours, 3 minutes. From preliminary analysis, the root cause of the issue is a configuration change that resulted in an overload of an internal component responsible for handling part of the traffic requests.
Customer Impact:
Increased TLS handshake errors or connection terminations for GCLB traffic served from HTTPS load balancers in Western Europe.
Additional details:
The issue was fully resolved on 27 June 2021 21:05 US/Pacific, and we are confident that there will not be a recurrence.
05:05 AM
The issue with HTTP load balancing has been resolved for all affected users in Europe as of Tuesday, 2021-07-27 19:02 US/Pacific.
We thank you for your patience while we worked on resolving the issue.
05:27 PM
Summary: HTTP load balancing returning error for some user in Europe.
Description: We believe the issue with Cloud Networking is resolved and there is no further impact.
Engineering team has mitigated the current impact and expect the full resolution to be rolled out within next 12 hours.
We will provide an update by Tuesday, 2021-07-27 21:30 US/Pacific with current details.
Diagnosis: None at this time
Workaround: None at this time
04:24 PM
Summary: HTTP load balancing returning error for some user in Europe.
Description: We believe the issue with Cloud Networking is resolved.
We do not have an ETA for full resolution at this point.
We will provide an update by Tuesday, 2021-07-27 15:30 US/Pacific with current details.
Diagnosis: none at this time
Workaround: none at this time
04:22 PM
Summary: HTTP load balancing returning error for some user in Europe.
Description: Mitigation work is currently underway by our engineering team.
We do not have an ETA for mitigation at this point.
We will provide more information by Tuesday, 2021-07-27 09:57 US/Pacific.
Diagnosis: none at this time
Workaround: none at this time
03:51 PM
Summary: HTTP load balancing returning error for some user in Europe.
Description: We are experiencing an intermittent issue with Cloud Networking beginning at Tuesday, 2021-07-27 06:56:46 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Tuesday, 2021-07-27 08:20 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: none at this time
Workaround: none at this time