Global: Increased rate of 500s and latency in L7ILB / rL7XLB Envoy

The issue with Google Cloud Networking has been resolved for all affected projects as of Monday, 2023-03-20 09:46 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Global: Increased rate of 500s and latency in L7ILB / rL7XLB Envoy Description: We believe the issue with Google Cloud Networking is partially resolved. Rollback has been completed in all affected regions, except: us-west3 Full resolution is expected to complete by Monday, 2023-03-20 10:00 US/Pacific. We will provide an update by Monday, 2023-03-20 10:30 US/Pacific with current details. Diagnosis: 500s responses returned from Envoy load balancers. Additional latency on queries passing through the load balancer and receive query errors depending on backend timeouts. Workaround: Customers can increase timeouts in their LB to reduce the impact.

Summary: Global: Increased rate of 500s and latency in L7ILB / rL7XLB Envoy Description: We believe the issue with Google Cloud Networking is partially resolved. Rollback has been completed in all affected regions, except: australia-southeast2 europe-west4 southamerica-west1 us-west3 Full resolution is expected to complete by Monday, 2023-03-20 09:00 US/Pacific. We will provide an update by Monday, 2023-03-20 09:30 US/Pacific with current details. Diagnosis: 500s responses returned from Envoy load balancers. Additional latency on queries passing through the load balancer and receive query errors depending on backend timeouts. Workaround: Customers can increase timeouts in their LB to reduce the impact.

Summary: Global: Increased rate of 500s and latency in L7ILB / rL7XLB Envoy Description: We believe the issue with Google Cloud Networking is partially resolved. Rollback has been completed in all affected regions, except: australia-southeast2 europe-west4 southamerica-west1 us-west3 Full resolution is expected to complete by Monday, 2023-03-20 09:00 US/Pacific. We will provide an update by Monday, 2023-03-20 09:00 US/Pacific with current details. Diagnosis: 500s responses returned from Envoy load balancers. Additional latency on queries passing through the load balancer and receive query errors depending on backend timeouts. Workaround: Customers can increase timeouts in their LB to reduce the impact.

Summary: Global: Increased rate of 500s and latency in L7ILB / rL7XLB Envoy Description: Engineering team has mitigated the issue in asia-south1 region and currently working on rollback for all global regions starting at 2023-03-17 12:00 US/Pacific. The ETA for the global rollback is Monday, 2023-03-20 08:30 US/Pacific We will provide more information by Monday, 2023-03-20 08:30 US/Pacific. Diagnosis: 500s responses returned from Envoy load balancers. Additional latency on queries passing through the load balancer and receive query errors depending on backend timeouts. Workaround: Customers can increase timeouts in their LB to reduce the impact.

Summary: Global: Increased rate of 500s and latency in L7ILB / rL7XLB Envoy Description: Engineering team has mitigated the issue in asia-south1 region and currently working on rollback for all global regions starting at 2023-03-17 12:00 US/Pacific. The ETA for the global rollback is Monday, 2023-03-20 05:30 US/Pacific We will provide more information by Monday, 2023-03-20 05:30 US/Pacific. Diagnosis: 500s responses returned from Envoy load balancers. Additional latency on queries passing through the load balancer and receive query errors depending on backend timeouts. Workaround: Customers can increase timeouts in their LB to reduce the impact.

Summary: Global: Increased rate of 500s and latency in L7ILB / rL7XLB Envoy Description: Engineering team has mitigated the issue in asia-south1 region and currently working on rollback for all global regions starting at 2023-03-17 12:00 US/Pacific. The ETA for the global rollback is Monday, 2023-03-20 03:00 US/Pacific We will provide more information by Monday, 2023-03-20 03:30 US/Pacific. Diagnosis: 500s responses returned from Envoy load balancers. Additional latency on queries passing through the load balancer and receive query errors depending on backend timeouts. Workaround: Customers can increase timeouts in their LB to reduce the impact.

Summary: Global: Increased rate of 500s and latency in L7ILB / rL7XLB Envoy Description: Engineering team has mitigated the issue in asia-south1 region and currently working on rollback for all global regions starting at 2023-03-17 12:00 US/Pacific. The ETA for the global rollback is Sunday, 2023-03-19 20:00 US/Pacific We will provide more information by Sunday, 2023-03-19 20:30 US/Pacific. Diagnosis: 500s responses returned from Envoy load balancers. Additional latency on queries passing through the load balancer and receive query errors depending on backend timeouts. Workaround: Customers can increase timeouts in their LB to reduce the impact.

Summary: Global: Increased rate of 500s and latency in L7ILB / rL7XLB Envoy Description: Engineering team has mitigated the issue in asia-south1 region and currently working on rollback for all global regions starting at 2023-03-17 12:00 US/Pacific. The ETA for the rollback is not known at this time. We will provide more information by Friday, 2023-03-17 15:30 US/Pacific. Diagnosis: 500s responses returned from Envoy load balancers. Additional latency on queries passing through the load balancer and receive query errors depending on backend timeouts. Workaround: Customers can increase timeouts in their LB to reduce the impact.

Summary: Global: Increased rate of 500s and latency in L7ILB / rL7XLB Envoy Description: Engineering team has mitigated the issue in asia-south1 region and currently working on rollback for all global regions starting at 2023-03-17 12:00 US/Pacific. The ETA for the rollback is not known at this time. We will provide more information by Friday, 2023-03-17 14:00 US/Pacific. Diagnosis: 500s responses returned from Envoy load balancers. Additional latency on queries passing through the load balancer and receive query errors depending on backend timeouts. Workaround: Customers can increase timeouts in their LB to reduce the impact.