We are experiencing issues in App Engine Flexible and Cloud Composer

Mini Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support . Affected Services and Features: Google App Engine - App Engine Flexible Google Cloud Composer Regions/Zones: global Description: Google App Engine (GAE) Flexible and Google Cloud Composer operations (i.e. create, update, delete operations) had intermittent failures for customers for a total duration of 10 hours, 30 minutes globally. From preliminary analysis, the root cause of the issue wass slow response to deployment workflows from one of our data centers hosting a backend component that manages deployments. The first two occurrences were self-solved by our automatic processes. While the investigations continued, the issue recurred multiple times. and the potential cause was identified and the traffic was directed away from the slow responding data center mitigating the issue on 09 November 2022 at 08:37 US/Pacific. Detailed information on occurrences: (All Times US/Pacific) First Occurrence : Impact Start: 08 November 2022 14:15 Impact End: 08 November 2022 15:05 Impact Duration: 50 minutes Second Occurrence : Impact Start: 08 November 2022 16:40 Impact End: 08 November 2022 19:37 Impact Duration: 2 hours, 57 minutes Third Occurrence : Impact Start: 09 November 2022 01:44 Impact End: 09 November 2022 08:27 Impact Duration: 6 hour, 43 minutes Total Duration: 10 hours, 30 minutes Customer Impact: Google App Engine Flexible : 16% of GAE flexible projects may have experienced issues with create, update, delete operations. Cloud Composer : Composer environment creation operations may have failed, typically with an error that mentions App Engine.

The issue with Google App Engine, Google Cloud Composer has been resolved for all affected projects as of Tuesday, 2022-11-08 20:20 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: We are experiencing issues in App Engine Flexible and Cloud Composer Description: We are experiencing an issue with Google App Engine, Google Cloud Composer. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2022-11-08 22:03 US/Pacific with current details. Diagnosis: App Engine Flexible : Customers may experience App Engine Flexible deployment issues with create/update/delete operations. Cloud Composer : Composer environment creation operations may fail, typically with an error that mentions App Engine Workaround: Cloud Composer : Customers can retry the operation after it fails, although there is a chance it may fail again.

Summary: App Engine Flexible deployment failures Description: We are experiencing an issue with Google App Engine beginning at Tuesday, 2022-11-08 14:15 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2022-11-08 20:55 US/Pacific with current details. Diagnosis: Customers may experience App Engine Flexible deployment issues with create/update/delete operations. Workaround: None