Global: Multiple Cloud Products Experiencing Deployment Failures

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support (All Times US/Pacific) Incident Start: 4 September, 2024 08:16 Incident End: 4 September, 2024 12:25 Duration: 4 hours, 9 minutes Affected Services and Features: Cloud Run, Google App Engine, Google Cloud Composer, Google Cloud Console, Google Cloud Functions, Google Cloud Marketplace, Cloud Deployment Manager Regions/Zones: Global Description: Globally, multiple cloud products experienced deployment failures due to the inaccessibility of Cloud Deployment Manager (CDM) services. This resulted in outages affecting all deployments and the CDM user interface within the cloud console for a duration of 4 hours and 9 minutes. From preliminary analysis, the root cause of the issue was a regressive change in our rollout strategy. This change inadvertently caused all incoming requests to be discarded, resulting in widespread RPC errors for both internal systems and external customers. Customer Impact: Cloud Composer: All Composer 1 operations (creations/updates/deletions) were failing globally. Running Composer 1 environments were not affected. All other versions (Composer 2, Composer 3) were not affected. Cloud Marketplace: Customers experienced errors when deploying Deployment Manager based solutions on Google Cloud Marketplace. Cloud Deployment Manager services were unreachable, caused outages for all deployments and the Deployment Manager UI in the Cloud Console. App Engine Flexible: App Engine Flexible experienced failures in delete flows in us-central1. 70-100% of all delete requests failed for ~90 min. spread across when the DM was unavailable. Serving plane was not affected. Serverless VPC Access: App Engine Standard, Cloud Functions, and Cloud Run were not able to create, update or delete VPC Access Connectors globally. Existing connectors were not affected. Cloud Deployment Manager: All Cloud Deployment Manager operations (create/update/deletes) were failing globally Customers were unable to view, update or delete the deployments via the Google Cloud Console All beta functionalities of Cloud Deployment Manager (Type Providers, Actions) were failing globally.

The issue with Cloud Run, Google App Engine, Google Cloud Composer, Google Cloud Functions, Google Cloud Marketplace, Firebase Extensions has been resolved for all affected users as of Wednesday, 2024-09-04 12:25 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Global: Multiple Cloud Products Experiencing Deployment Failures Description: We are experiencing an issue with Google Cloud Composer, Google Cloud Marketplace, App Engine Flexible, Cloud Run, App Engine Flexible, and Cloud Functions beginning on Wednesday, 2024-09-04 09:21 US/Pacific. Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide an update by Wednesday, 2024-09-04 13:00 US/Pacific with current details. Diagnosis: Cloud Composer: All Composer 1 operations (creations/updates/deletions) are failing globally. Running Composer 1 environments are not affected. All other versions (Composer 2, Composer 3) are not affected. Cloud Marketplace Customers experience errors when deploying Deployment Manager based solutions on Google Cloud Marketplace. Cloud Deployment Manager services are unreachable, causing outages for all deployments and the Deployment Manager UI in the Cloud Console. App Engine Flexible App Engine Flexible new version deployments are experiencing failures in us-central1. Serverless VPC Access App Engine Standard, Cloud Functions, and Cloud Run will not be able to create VPC Access connectors. Existing connectors are not affected. Workaround: Affected Google Cloud Composer customers should avoid running any operations on Cloud Composer 1 environments. There are no workarounds for Deployment Manager failures.

Summary: Global: Multiple Cloud Products Experiencing Deployment Failures Description: We are experiencing an issue with Google Cloud Composer, Google Cloud Marketplace, and App Engine Flexible beginning on Wednesday, 2024-09-04 09:21 US/Pacific. Our engineering team continues to work on identifying a mitigation strategy. We will provide an update by Wednesday, 2024-09-04 12:40 US/Pacific with current details. Diagnosis: Cloud Composer: All Composer 1 operations (creations/updates/deletions) are failing globally. Running Composer 1 environments are not affected. All other versions (Composer 2, Composer 3) are not affected. Cloud Marketplace Customers experience errors when deploying Deployment Manager based solutions on Google Cloud Marketplace. Cloud Deployment Manager services are unreachable, causing outages for all deployments and the Deployment Manager UI in the Cloud Console. App Engine Flexible App Engine Flexible new version deployments are experiencing failures in us-central1. Workaround: - Affected Google Cloud Composer customers should avoid running any operations on Cloud Composer 1 environments. There are no workarounds for Deployment Manager failures.