MemoryStore for Redis - instances update/export failures

Mini Incident Report We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 26 September 2022 05:58 Incident End: 03 October 2022 09:00 Duration: 7 days, 3 hours, 2 minutes Affected Services and Features: Cloud MemoryStore Regions/Zones: Regional Description: On 26 September 2022, 05:58 AM US/Pacific Cloud Memory Store experienced failures and errors when attempting to start, update, scale or export instances. From preliminary investigation the root cause appears to be due to a configuration version mismatch, as a result instances operations were never started and would timeout. Google implemented an update to fix the configuration mismatch which restored service and the issue was mitigated 03 October 2022 09:00 US/Pacific. Additional details: Some basic and standard tier instances experienced instance failures like scaling or export with timeouts errors.

The issue with Cloud Memorystore has been resolved for all affected users as of Monday, 2022-10-03 09:15 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is currently underway by our engineering team. Most of the affected instances have recovered. We do not have an ETA for mitigation completion at this point. We will provide more information by Wednesday, 2022-10-05 12:00 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is currently underway by our engineering team. Most of the affected instances have recovered. We do not have an ETA for mitigation completion at this point. We will provide more information by Friday, 2022-09-30 12:00 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is currently underway by our engineering team. Most of the affected instances have recovered. We do not have an ETA for mitigation completion at this point. We will provide more information by Friday, 2022-09-30 10:30 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is currently underway by our engineering team. Most of the affected instances have recovered. We do not have an ETA for mitigation completion at this point. We will provide more information by Thursday, 2022-09-29 15:30 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is currently underway by our engineering team. Most of the affected instances have recovered. We do not have an ETA for mitigation completion at this point. We will provide more information by Thursday, 2022-09-29 12:30 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is currently underway by our engineering team. Most of the affected instances have recovered. We do not have an ETA for mitigation completion at this point. We will provide more information by Thursday, 2022-09-29 11:30 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is currently underway by our engineering team. Most of the affected instances have recovered. We do not have an ETA for mitigation completion at this point. We will provide more information by Thursday, 2022-09-29 10:30 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2022-09-28 19:00 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is underway by our engineering team. We will provide more information by Wednesday, 2022-09-28 17:00 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is underway by our engineering team. We will provide more information by Wednesday, 2022-09-28 14:30 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is underway by our engineering team. We will provide more information by Wednesday, 2022-09-28 13:30 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is underway by our engineering team. We will provide more information by Wednesday, 2022-09-28 12:30 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is underway by our engineering team. We will provide more information by Wednesday, 2022-09-28 11:30 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is underway by our engineering team. We will provide more information by Wednesday, 2022-09-28 09:30 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is underway by our engineering team. We will provide more information by Tuesday, 2022-09-27 17:30 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is underway by our engineering team. We will provide more information by Tuesday, 2022-09-27 15:10 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.

Summary: MemoryStore for Redis - instances update/export failures Description: Mitigation work is underway by our engineering team. We will provide more information by Tuesday, 2022-09-27 15:00 US/Pacific. Diagnosis: Customers may see failures while attempting to update/export their basic tier instances. Workaround: None at this time.