We are experiencing an issue with Cloud Memorystore, AlloyDB for PostgreSQL, Backup and DR, Cloud Data Fusion, Googl...

Mini Incident Report We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support (All Times US/Pacific) Incident Start: 20 November 2023 22:29 Incident End: 21 November 2023 03:07 Duration: 4 hours, 38 minutes Affected Services and Features: Cloud SQL AlloyDB Cloud Data Fusion Cloud Composer Cloud Backup and DR Cloud Memorystore Regions/Zones: Global Description: Multiple Google Cloud products experienced elevated latency and error rates for a duration of 4 hours, 38 minutes in the launch of new producer service instances (like Cloud SQL, etc.) This did not impact existing instances. From preliminary analysis, the root cause of the issue was an unexpected load that caused a surge in API requests to Service Networking, which led to increased latency for these requests. Requests to create Service Networking from consumers to create connections, or to producer services to launch service instances, which in turn relies on Service Networking, slowed down. The service networking consumer-side API experienced up to 10 times the normal levels of create connection requests, and up to 3.5 times the normal levels of add subnetwork requests. While our engineering team was working on troubleshooting the issue, the surge started decreasing at around 01:25 US/Pacific and once the requests were back to normal, the service recovered at 03:07 US/Pacific Customer Impact: Cloud SQL: CREATE operations that require a subnet creation in the tenant project were slow, and took up to one hour to complete. AlloyDB: Instance creations failed with timeouts due to service networking creation taking more than 10 minutes. Cloud Data Fusion: Private Data Fusion instance creation failed or timed out due to Service Networking and Cloud SQL creation. Cloud Composer: Environment creations fail in multiple regions due to failed Cloud SQL creations. Cloud Backup and DR: ManagementServer creations failed globally for the duration of the outage. . Cloud Memorystore: ServiceNetwork instance creation experienced elevated latency and failures.

Google Cloud SQL Impact: CREATE operations that require a subnet creation in the tenant project are slow, and can take up to one hour to complete. Mitigated as of 2023-11-21 02:12:09 US/Pacific AlloyDB Impact: Instance creations fail with timeouts due to service networking creation taking more than 10 minutes. Mitigated as of 2023-11-21 02:30:31 US/Pacific Cloud Data Fusion Impact: Private Data Fusion instance creation failing/timing out due to Service Networking and Cloud SQL creation. Mitigated as of 2023-11-21 02:28:10 US/Pacific Cloud Composer Impact: Environment creations fail in multiple regions due to failed Cloud SQL creations. Mitigated as of 2023-11-21 02:28:23 US/Pacific Cloud Backup and DR Impact: All ManagementServer creations are failing globally. Mitigated as of 2023-11-21 02:37:04 US/Pacific MemoryStore Impact: ServiceNetwork operations may be slow. Mitigated as of 2023-11-21 02:38:00 US/Pacific We thank you for your patience while we worked on resolving the issue.

Summary: We are experiencing an issue with Cloud Memorystore. AlloyDB for PostgreSQL, Backup and DR, Cloud Data Fusion, Google Cloud Composer, Google Cloud SQL are mitigated Description: Mitigated: Google Cloud SQL Impact: CREATE operations that require a subnet creation in the tenant project are slow, and can take up to one hour to complete. Mitigated as of 2023-11-21 02:12:09 US/Pacific AlloyDB Impact: Instance creations fail with timeouts due to service networking creation taking more than 10 minutes. Mitigated as of 2023-11-21 02:30:31 US/Pacific Cloud Data Fusion Impact: Private Data Fusion instance creation failing/timing out due to Service Networking and Cloud SQL creation. Mitigated as of 2023-11-21 02:28:10 US/Pacific Cloud Composer Impact: Environment creations fail in multiple regions due to failed Cloud SQL creations. Mitigated as of 2023-11-21 02:28:23 US/Pacific Cloud Backup and DR Impact: All ManagementServer creations are failing globally. Mitigated as of 2023-11-21 02:37:04 US/Pacific Ongoing impact: MemoryStore Impact: ServiceNetwork operations may be slow. We will provide an update by Tuesday, 2023-11-21 03:15 US/Pacific with current details. Diagnosis: Creation of resources related to Cloud Service Networking may experience failures or delays. Workaround: Attempt retry or await for the operation to complete.

Summary: We are experiencing an issue with AlloyDB for PostgreSQL, Backup and DR, Cloud Data Fusion, Cloud Memorystore, Google Cloud Composer, Google Cloud SQL Description: We are experiencing an issue with AlloyDB for PostgreSQL, Backup and DR, Cloud Data Fusion, Cloud Memorystore, Google Cloud Composer, Google Cloud SQL. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2023-11-21 04:27 US/Pacific with current details. Diagnosis: Creation of resources related to Cloud Service Networking may experience failures or delays. Workaround: Attempt retry or await for the operation to complete.

Summary: We are experiencing an issue with AlloyDB for PostgreSQL, Backup and DR, Cloud Data Fusion, Cloud Memorystore, Google Cloud Composer, Google Cloud SQL Description: We are experiencing an issue with AlloyDB for PostgreSQL, Backup and DR, Cloud Data Fusion, Cloud Memorystore, Google Cloud Composer, Google Cloud SQL beginning at Monday, 2023-11-20 22:30 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2023-11-21 02:40 US/Pacific with current details. Diagnosis: None at this time. Workaround: None at this time.