Cloud Composer 2 customers experiencing create, update and snapshot operation failures in multiple regions.

Mini Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support . (All Times US/Pacific) Incident Start: 1 August 2023 at 23:00 Incident End: 2 August 2023 at 03:45 Duration: 4 hours, 45 minutes Affected Services and Features: Cloud Composer Regions/Zones: asia-east1 asia-east2 asia-northeast1 asia-northeast2 asia-northeast3 australia-southeast2 europe-west2 us-central1 europe-west1 europe-west9 Description: Cloud Composer 2 environments experienced creation, update, and snapshots operation failures for a duration of 4 hours and 45 minutes in multiple regions. From preliminary analysis, the root cause was the invalid proxy configuration, which blocked outgoing traffic to Google Kubernetes Engine (GKE). Customer Impact: Customers experienced failures when attempting the following actions: Create new Composer 2 environment Update existing Composer 2 environment Save and load Composer 2 environment snapshots DAGs executions were not impacted by the outage as long as configuration changes were not being introduced.

The issue with Google Cloud Composer has been resolved for all affected projects as of Wednesday, 2023-08-02 04:49 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Cloud Composer 2 customers experiencing create, update and snapshot operation failures in multiple regions. Description: Mitigation work is currently underway and we believe the issue is mostly resolved at this time. We're performing checks to confirm and we will provide an update by Wednesday, 2023-08-02 06:00 US/Pacific. Diagnosis: Cloud Composer 2 customers impacted by this issue may see create and/or update and snapshot operation failing. Workaround: Customers are recommended to not perform any update operations until this incident is over and refrain from creating Composer environments in the impacted regions. DAG executions in existing environments should work fine assuming that the customer doesn’t try to introduce changes related to Airflow components reconfiguration.

Summary: Cloud Composer 2 customers experiencing create, update and snapshot operation failures in multiple regions: asia-east1 asia-east2 asia-northeast1 asia-northeast2 asia-northeast3 australia-southeast2 europe-southwest1 europe-west1 europe-west2 europe-west9 me-central1 southamerica-west1 us-central1 Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-08-02 05:00 US/Pacific. Diagnosis: Cloud Composer 2 customers impacted by this issue may see create and/or update and snapshot operation failing. Workaround: Customers are recommended to not perform any update operations until this incident is over and refrain from creating Composer environments in the impacted regions. DAG executions in existing environments should work fine assuming that the customer doesn’t try to introduce changes related to Airflow components reconfiguration.

Summary: Cloud Composer 2 customers experiencing create and update operation failures in multiple regions: asia-east1 asia-east2 asia-northeast1 asia-northeast2 asia-northeast3 australia-southeast2 europe-west2 me-central1 us-central1 europe-southwest1 europe-west1 europe-west9   - southamerica-west1 Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-08-02 04:00 US/Pacific. Diagnosis: Cloud Composer 2 customers impacted by this issue may see create and/or update operation failing Workaround: None at this time.

Summary: Cloud Composer 2 customers experiencing create and update operation failures in multiple regions Description: We are experiencing an issue with Google Cloud Composer 2 beginning on Wednesday, 2023-08-01 22:30 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2023-08-02 03:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Cloud Composer 2 customers impacted by this issue may see create and/or update operation failing Workaround: None at this time.