Google Cloud MINOR
MDP (an istio component) evicts pods continuously when the image URL is overwritten by Kubernetes-digester for versio...
November 21, 2023 · 03:43 PM UTC – 11:33 PM UTC · Duration: 7h 50min
Affected Services
Anthos Service Mesh
Timeline
11:33 PM
The issue with Anthos Service Mesh interacting with k8s-digester has an impact on a very small number of projects having a fully managed GCP service.
We sincerely apologize for any inconvenience this may cause. Please rest assured that our engineering teams are working diligently to mitigate all known impacts.
In the meantime, if you have any questions or are impacted by this issue, please open a case with our support team, and we will work with you until this issue is resolved.
** Please go to https://console.cloud.google.com/support/cases for more information on how to open a support case.**
We sincerely appreciate your patience and understanding as we work to resolve it as quickly as possible.
07:33 PM
Summary: MDP (an istio component) evicts pods continuously when the image URL is overwritten by Kubernetes-digester for versions of istio: 1.14.6-asm.23 & 1.14.6-asm.24
Description: We are experiencing an issue with Anthos Service Mesh beginning at Tuesday, 2023-11-21 06:39 US/Pacific.
Our engineering team continues to investigate the issue. we sincerely apologize for any inconvenience this may cause. Please rest assured that our engineering teams are working diligently to mitigate all known impacts.
We will provide an update by Tuesday, 2023-11-21 20:00 US/Pacific with current details.
Diagnosis: Customers may experience their Anthos Service Mesh (ASM) pods being continuously (and sometimes simultaneously) evicted when the proxy image URL is overwritten by Kubernetes-digester.
Workaround: As a workaround, we suggest that:-
Customers should disable MDP if they are impacted. Instruction how to disable MDP is available here:
https://cloud.google.com/service-mesh/docs/managed/provision-managed-anthos-service-mesh#disable_the_managed_data_plane_optional
** Customers disable Managed ASM individually on each namespace with the following commands:
kubectl annotate --overwrite controlplanerevision -n istio-system
REVISION_LABEL mesh.cloud.google.com/proxy='{"managed":"false"}'
04:18 PM
Summary: MDP (an istio component) evicts pods continuously if image url is overwritten by k8s-digester for versions of istio: 1.14.6-asm.23 & 1.14.6-asm.24
Description: We've received a report of an issue with Anthos Service Mesh as of Tuesday, 2023-11-21 06:39 US/Pacific.
We will provide more information by Tuesday, 2023-11-21 12:00 US/Pacific.
Diagnosis: Anthos Service Mesh (ASM) pods are continuously (and sometimes simultaneously) evicted if the proxy image url is overwritten by k8s-digester.
Workaround: Customers should disable MDP if they are impacted. Instruction how to disable MDP is available here:
https://cloud.google.com/service-mesh/docs/managed/provision-managed-anthos-service-mesh#disable_the_managed_data_plane_optional
Please disable Managed ASM individually on each namespace with the following commands:
kubectl annotate --overwrite controlplanerevision -n istio-system
REVISION_LABEL mesh.cloud.google.com/proxy='{"managed":"false"}'
03:53 PM
Summary: MDP (an istio component) evicts pods continuously if image url is overwritten by k8s-digester for versions of istio: 1.14.6-asm.23 & 1.14.6-asm.24
Description: We've received a report of an issue with Anthos Service Mesh as of Tuesday, 2023-11-21 06:39 US/Pacific.
We will provide more information by Wednesday, 2023-11-29 10:00 US/Pacific.
Diagnosis: Anthos Service Mesh (ASM) pods are continuously (and sometimes simultaneously) evicted if the proxy image url is overwritten by k8s-digester.
Workaround: Customers should disable MDP if they are impacted. Instruction how to disable MDP is available here:
https://cloud.google.com/service-mesh/docs/managed/provision-managed-anthos-service-mesh#disable_the_managed_data_plane_optional
Please disable Managed ASM individually on each namespace with the following commands:
kubectl annotate --overwrite controlplanerevision -n istio-system
REVISION_LABEL mesh.cloud.google.com/proxy='{"managed":"false"}'