Global: Some Google Cloud Console Customers are unable to create/edit GCE instances

SUMMARY: On Tuesday, 14 June 2022, customers were unable to create or edit Google Compute Engine (GCE) instances via the Google Cloud Console for 2 days, 8 hours, 27 minutes. To our customers that were impacted during this outage, we sincerely apologize. We are conducting an internal investigation and are taking steps to improve our service. ROOT CAUSE: Customers can specify organization policies to limit what instances can use external IP addresses (compute.vmExternalIpAccess). Setting an IP address on an instance not allowed by policy will cause an operation failure. The incident was triggered by a compute frontend UI release which made it impossible for certain users to modify instances due to interactions between org policies and a bug that forced an Ephemeral IP address while on either the edit or create page. Any user with the compute.vmExternalIpAccess policy could not create or edit instances without a public IP. A bug was identified where customers with a policy restricting external IP addresses were not able to select one during instance creation. An attempt to fix this bug created a regression where changing any field in the Edit instance page would change the IP address to ephemeral for instances that had no IP address selected. Because of the org policy blocks assigning the instance a public IP, the save operation would fail.The release containing this change included a fix to address the bug however, once in production several customers reported issues. REMEDIATION AND PREVENTION: Google engineers were alerted to the issue via customer support case on Thursday, 16 June 2022 at 04:15 and started an investigation. At 06:11, Google engineers were able to reproduce the issue and escalated the incident at 09:08. At 09:50, Google engineers initiated a bug of the release which was completed at 11:33 fully mitigating the issue. Google is committed to improving our service in the future and will be completing the following actions: Improve unit testing for org policies to identify issues of this type. Improve alerting to quickly detect configuration failures. DETAILED DESCRIPTION OF IMPACT: On Tuesday, 14 June 2022 03:06 to Thursday 16 June 11:33 US/Pacific Google Compute Engine Affected customers experienced failures creating or editing GCE instances via the Google Cloud Console and may have received an error “Constraint constraints/compute.vmExternalIpAccess violated for project [project ID].“ ADDITIONAL INFORMATION FOR CUSTOMERS: As a workaround, customers were still able to create or edit GCE instances via the gcloud CLI or via Google Cloud Console by disabling the constraints/compute.vmExternalIpAccess policy.

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 13 June 2022 14:55 Incident End: 16 June 2022 11:26 Duration: 2 days, 20 hours, 31 minutes Affected Services and Features: Google Cloud Console, Google Compute Engine Regions/Zones: Global Description: Customers may have been unable to create or edit Google Compute Engine (GCE) instances via the Google Cloud Console for 2 days 20 hours 31 minutes. From preliminary analysis, the root cause of the issue was a recent update to the GCE frontend UI. Google engineers rolled back the GCE frontend UI update to mitigate the issue on 16 June 2022 11:33 US/Pacific. Customer Impact: Customers attempting to create or edit GCE instances via the Google Cloud Console may have received an error “Constraint constraints/compute.vmExternalIpAccess violated for project [project ID].“ As a workaround, customers were still able to create or edit GCE instances via the gcloud CLI or via Google Cloud Console by disabling the constraints/compute.vmExternalIpAccess policy.

The issue with Google Cloud Console, Google Compute Engine has been resolved for all affected projects as of Thursday, 2022-06-16 11:33 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Global: Some Google Cloud Console Customers are unable to create/edit GCE instances Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2022-06-16 12:00 US/Pacific. Diagnosis: Affected customers are unable to create/edit GCE instances via Cloud Console. Workaround: 1) Using gcloud CLI 2) Disabling the policy of constraints/compute.vmExternalIpAccess

Summary: Global: Some Google Cloud Console Customers are unable to create/edit GCE instances Description: We are experiencing an issue with Google Compute Engine, Google Cloud Console beginning at Thursday, 2022-06-16 09:08 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-06-16 10:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers are unable to create/edit GCE instances via Cloud Console. Workaround: 1) Using gcloud CLI 2) Disabling the policy of constraints/compute.vmExternalIpAccess

Summary: Global: Some Google Cloud Console Customers are able to edit GCE instances Description: We are experiencing an issue with Google Compute Engine, Google Cloud Console beginning at Thursday, 2022-06-16 09:08 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-06-16 09:55 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers are unable to create/edit GCE instances via Cloud Console. Workaround: 1) Using gcloud CLI 2) Disabling the policy of constraints/compute.vmExternalIpAccess

Summary: Global: Some Google Cloud Console Customers are able to edit GCE instances Description: We are experiencing an issue with Google Compute Engine, Google Cloud Console beginning at Thursday, 2022-06-16 09:08 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-06-16 09:55 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers are unable to edit any GCE instances via Cloud Console. Workaround: None at this time.