Google Cloud MINOR

Global: Media CDN experiencing elevated playback failures

July 18, 2023 · 08:18 PM UTC – 04:12 AM UTC · Duration: 55h 54min

Affected Services

Media CDNGoogle Cloud Networking

Timeline

04:12 AM
The issue with Media CDN has been resolved for all affected users as of Thursday, 2023-07-20 16:00 US/Pacific. We thank you for your patience while we worked on resolving the issue.
11:15 PM
Summary: Global: Media CDN experiencing elevated playback failures Description: The fix roll out is completed and we believe the issue with Media CDN is fully mitigated at this point. Our Engineers are working closely with a subset of customers to verify resolution of the issue. We will provide an update by Thursday, 2023-07-20 20:30 US/Pacific with current details. Diagnosis: Customers may see a small percentage of connections being reset, which may manifest as playback failures. Workaround: None at this time.
06:20 PM
Summary: Global: Media CDN experiencing elevated playback failures Description: Our engineering team identified a potential root cause and is working on a mitigation strategy. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2023-07-20 15:00 US/Pacific. Diagnosis: Customers may see a small percentage of connections being reset, which may manifest as playback failures. Workaround: None at this time.
10:08 PM
Summary: Global: Media CDN experiencing elevated playback failures Description: Our engineering team identified a potential root cause and is working on a mitigation strategy. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-07-19 15:00 US/Pacific. Diagnosis: Customers may see a small percentage of connections being reset, which may manifest as playback failures. Workaround: None at this time.
08:57 PM
Summary: Global: Media CDN experiencing elevated playback failures Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Tuesday, 2023-07-18 15:00 US/Pacific. Diagnosis: Customers may see a small percentage of connections being reset, which may manifest as playback failures. Workaround: None at this time.
08:36 PM
Summary: Global: Media CDN experiencing elevated playback failures Description: We are experiencing an issue with Media CDN beginning at Wednesday, 2023-07-12 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2023-07-18 13:07 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may see a small percentage of connections being reset, which may manifest as playback failures. Workaround: None at this time.