Vertex AI Search seeing elevated failures for 50% of queries for SearchService

Mini Incident Report We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support (All Times US/Pacific) Incident Start: 01 October, 2024 09:28 Incident End: 01 October, 2024 10:03 Duration: 35 minutes Affected Services and Features: Vertex AI Search for Retail Regions/Zones: Global Description: Vertex AI Search for Retail queries on the Search API experienced elevated error rates and latencies for a duration of 35 minutes. Preliminary analysis indicates the root cause is an issue introduced by a new version release of an internal service which the Search API relies on which caused failures on our Search API. The issue was mitigated by rolling out a patch for our internal service to resolve the disruption. Customer Impact: Vertex AI Search for Retail customers experienced elevated error rates and elevated latencies for a subset of queries. Additional Information: At the time of the outage, we believed that this issue was also impacting Recommendation AI. Upon further investigation, we concluded that this issue had no impact on Recommendation AI.

We experienced an issue with Vertex AI Search, Recommendation AI beginning on Tuesday, 2024-10-01 09:29 US/Pacific. Our monitoring indicates that the errors have stopped occurring as of 2024-10-01 10:05 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Vertex AI Search seeing elevated failures for 50% of queries for SearchService Description: We are experiencing an issue with Vertex AI Search, Recommendation AI beginning on Tuesday, 2024-10-01 09:29 US/Pacific. Our engineering team continues to investigate the issue with the highest priority We will provide an update by Tuesday, 2024-10-01 12:00 US/Pacific with current details. Diagnosis: Customers may observe elevated errors while performing search for products. Workaround: None at this time.

Summary: Vertex AI Search seeing elevated failures for 50% of queries for SearchService Description: We are experiencing an issue with Vertex AI Search, Recommendation AI beginning on Tuesday, 2024-10-01 09:29 US/Pacific. Our engineering team continues to investigate the issue with the highest priority We will provide an update by Tuesday, 2024-10-01 12:00 US/Pacific with current details. Diagnosis: Customers may observe elevated errors while performing search for products. Workaround: None at this time.

Summary: Vertex AI Search seeing elevated failures for 50% of queries for SearchService Description: We are experiencing an issue with Vertex AI Search, Recommendation AI beginning on Tuesday, 2024-10-01 09:29 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2024-10-01 11:45 US/Pacific with current details. Diagnosis: Customers may observe elevated errors while performing search for products. Workaround: None at this time.