Global: Cloud Scheduler Pub/Sub jobs fail with permission denied

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support (All Times US/Pacific) Incident Start: 21 July 2021 19:22 Incident End: 27 July 2021 15:28 Duration: 5 days, 20 hours, 6 minutes Affected Services and Features: Google Cloud Scheduler Pub/Sub Regions/Zones: All Regions Description: Google Cloud Scheduler jobs experienced increased errors globally when publishing messages to Pub/Sub topics for a duration of 5 days, 20 hours, 6 minutes. From preliminary analysis, the root cause of the issue is due to a configuration change that updated the service agent used when publishing to Pub/Sub. Projects using the new service agent without the correct permissions resulted in PERMISSION_DENIED errors for tasks that required publishing to Pub/Sub. Customer Impact: All customers with Cloud Scheduler jobs with a Pub/Sub topic as a target that did not grant the Cloud Scheduler Google-managed service account access to that Pub/Sub topic saw PERMISSION_DENIED errors. Additional details: The issue was fully resolved on 27 July 2021 at 15:28 US/Pacific after a rollback of the change was completed.

The issue with Cloud Scheduler has been resolved for all affected projects as of Tuesday, 2021-07-27 15:43 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: Global: Cloud Scheduler Pub/Sub jobs fail with permission denied Description: We believe the issue with Cloud Scheduler is partially resolved. We do not have an ETA for full resolution at this point. We will provide an update by Tuesday, 2021-07-27 16:01 US/Pacific with current details. Diagnosis: Receiving Cloud Scheduler PERMISSION_DENIED Workaround: Add the permission pubsub.topics.publish to Cloud Scheduler service account (service-PROJECT_NUMBER@gcp-sa-cloudscheduler.iam.gserviceaccount.com).

Summary: Global: Cloud Scheduler Pub/Sub jobs fail with permission denied Description: We believe the issue with Cloud Scheduler is partially resolved and there is no further impact observed. Action: Customers should utilize the Cloud Services Robot account for authentication. We will provide an update by Tuesday, 2021-07-27 15:30 US/Pacific with current details. Diagnosis: Receiving Cloud Scheduler PERMISSION_DENIED Workaround: Add the permission pubsub.topics.publish to Cloud Scheduler service account (service-PROJECT_NUMBER@gcp-sa-cloudscheduler.iam.gserviceaccount.com).

Summary: Global: Cloud Scheduler Pub/Sub jobs fail with permission denied Description: We believe the issue with Cloud Scheduler is partially resolved and there is no further impact observed. Action: Customers should utilize the Cloud Services Robot account for authentication. We will provide an update by Tuesday, 2021-07-27 13:30 US/Pacific with current details. Diagnosis: Receiving Cloud Scheduler PERMISSION_DENIED Workaround: Add the permission pubsub.topics.publish to Cloud Scheduler service account (service-PROJECT_NUMBER@gcp-sa-cloudscheduler.iam.gserviceaccount.com).

Summary: Global: Cloud Scheduler Pub/Sub jobs fail with permission denied Description: Mitigation work is still underway by our engineering team. The rollback activity is currently 50% complete and ongoing. We will provide more information by Tuesday, 2021-07-27 13:00 US/Pacific. Diagnosis: Receiving Cloud Scheduler PERMISSION_DENIED Workaround: Add publisher role to Cloud Scheduler service account. The service account has the form service-PROJECT_NUMBER@gcp-sa-cloudscheduler.iam.gserviceaccount.com

Summary: Global: Cloud Scheduler Pub/Sub jobs fail with permission denied Description: This is a continuation of the previous post for incident "Global: Cloud Scheduler Pub/Sub jobs fail with permission denied" that was closed as resolved. We have received updates from our engineering team that the Mitigation work is still underway for some regions and are currently waiting for an ETA. We will provide more information by Tuesday, 2021-07-27 11:00 US/Pacific. Diagnosis: Receiving Cloud Scheduler PERMISSION_DENIED Workaround: Add publisher role to Cloud Scheduler service account. The service account has the form service-PROJECT_NUMBER@gcp-sa-cloudscheduler.iam.gserviceaccount.com