[Global] Google App Engine - Customers using the Secrets Manager are experiencing increased errors.

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 08 June 2022 09:16 Incident End: 14 June 2022 20:02 Duration: 6 day, 10 hours, 46 minutes Affected Services and Features: Google Cloud App Engine, Google Cloud Functions. Regions/Zones: Global Description: Google Cloud App Engine customers might have experienced elevated errors while using the Secrets Manager for a duration of 6 days, 10 hours and 46 minutes. From preliminary analysis, the root cause of the issue is, regression in a recent software release relating to Secret Manager functionality in Google Cloud App Engine. Customer Impact: Customers might have experienced increased errors from apps or functions that use the Secrets Manager. Additional details: The Secret Manager functionality related release updates are rolled back to mitigate the issue. During the impact customers were advised to mount each secret in a separate path as per https://cloud.google.com/functions/docs/configuring/secrets.

The issue with Google App Engine has been resolved for all affected projects as of Tuesday, 2022-06-14 20:02 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: [Global] Google App Engine - Customers using the Secrets Manager are experiencing increased errors. Description: Mitigation work is still underway by our engineering team. We will provide more information by Tuesday, 2022-06-14 22:00 US/Pacific. Diagnosis: Customers using the Secrets Manager are experiencing increased errors. Workaround: A workaround is to mount each secret in a separate path. Please refer to https://cloud.google.com/functions/docs/configuring/secrets for information on Secret Manager.

Summary: [Global] Google App Engine - Customers using the Secrets Manager are experiencing increased errors. Description: Mitigation work is still underway by our engineering team. We will provide more information by Tuesday, 2022-06-14 18:00 US/Pacific. Diagnosis: Customers using the Secrets Manager are experiencing increased errors. Workaround: A workaround is to mount each secret in a separate path. Please refer to https://cloud.google.com/functions/docs/configuring/secrets for information on Secret Manager.

Summary: Increase in errors correlated with binary rollout Description: Mitigation work is currently underway by our engineering team. The mitigation is completed completed for europe-west3. The mitigation ETA for us-central1 is Tuesday, 2022-06-14 14:00 US/Pacific. ETA will be provided for other regions in a subsequent update. We will provide more information by Tuesday, 2022-06-14 14:30 US/Pacific. Diagnosis: Customers using Secrets Manager are experiencing increased errors. Workaround: A workaround is to mount each secret in a separate path. Please refer to https://cloud.google.com/functions/docs/configuring/secrets for information on Secret Manager.

Summary: Increase in errors correlated with binary rollout Description: Mitigation work is currently underway by our engineering team. The mitigation is expected to be completed for europe-west3 Tuesday, 2022-06-14 10:00 US/Pacific, and for us-central1 Tuesday, 2022-06-14 14:00 US/Pacific. ETA will be provided for other regions in a subsequent update. We will provide more information by Tuesday, 2022-06-14 10:10 US/Pacific. Diagnosis: Customers using Secrets Manager are experiencing increased errors. Workaround: A workaround is to mount each secret in a separate path. Please refer to https://cloud.google.com/functions/docs/configuring/secrets for information on Secret Manager.

Summary: Increase in errors correlated with binary rollout Description: Mitigation work is currently underway by our engineering team. The mitigation is expected to be completed for europe-west3 Tuesday, 2022-06-14 10:00 US/Pacific, and for us-central1 Tuesday, 2022-06-14 14:00 US/Pacific. ETA will be provided for other regions in a subsequent update. We will provide more information by Tuesday, 2022-06-14 08:27 US/Pacific. Diagnosis: Customers are experiencing increased errors. Workaround: None at this time.