Google Cloud MINOR

Multi-Region: Inconsistent Verify JWT policy execution

March 15, 2023 · 12:35 AM UTC – 10:19 AM UTC · Duration: 57h 44min

Affected Services

Apigee

Timeline

10:19 AM
The issue with Apigee has been resolved for all affected users as of Friday, 2023-03-17 02:15 US/Pacific. We thank you for your patience while we worked on resolving the issue.
02:10 AM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team. A rollback is still in progress in us-east1. Rollback has been completed and an issue is mitigated in: europe-north1 and us-west2. We do not have an ETA for mitigation at this point for us-east1. We will provide more information by Friday, 2023-03-17 12:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
05:43 PM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team. A rollback is still in progress in us-east1. Rollback has been completed and an issue is mitigated in: europe-north1 and us-west2. We do not have an ETA for mitigation at this point for us-east1. We will provide more information by Thursday, 2023-03-16 18:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
04:36 PM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team. A rollback is still in progress in us-east1. Rollback has been completed and an issue is mitigated in: europe-north1 and us-west2. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2023-03-16 10:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
11:58 AM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team. A rollback is in progress and engineers have completed rollbacks for 90% of the orgs affected. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2023-03-16 10:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
01:59 AM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team. A rollback is in progress and engineers have completed rollbacks for 90% of the orgs affected. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2023-03-16 06:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
08:07 PM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Our engineering team has determined that further investigation is required to mitigate the issue. We will provide an update by Wednesday, 2023-03-15 18:00 US/Pacific with current details. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
03:22 PM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-03-15 12:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
01:39 PM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-03-15 08:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
12:59 PM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-03-15 06:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
09:35 AM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-03-15 05:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
07:32 AM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-03-15 03:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
04:40 AM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-03-15 01:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
03:17 AM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Tuesday, 2023-03-14 21:30 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
02:13 AM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Tuesday, 2023-03-14 19:30 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
01:18 AM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Tuesday, 2023-03-14 18:30 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.
12:49 AM
Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: We are experiencing an issue with Apigee beginning at Tuesday, 2023-03-14 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2023-03-14 17:20 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time.