Windows VMs using CrowdStrike are crashing.

Beginning July 19th at 04:09 UTC, Google Cloud detected some customer Windows VMs experiencing Blue Screen of Death (BSOD) and crash loops. These Windows VMs running CrowdStrike Falcon began to fail after a CrowdStrike software update. Crowdstrike quickly deployed a fix, however some customer impact remained. While Google Cloud services were not directly impacted, Google Cloud continues to work with CrowdStrike to help our customers recover from any remaining impact. Crowdstrike has published a statement about this incident recommending steps for workarounds and remediation: https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/ If your Windows VM continues to experience issues after a reboot, manual patching. Please contact Google Cloud Customer Support.

Summary: Windows VMs using CrowdStrike are crashing. Description: We are experiencing an issue with Windows VMs running CrowdStrike on Google Compute Engine. CrowdStrike has confirmed that a faulty update to the CrowdStrike Falcon agent was deployed beginning at 04:09 UTC July 19. After having automatically received a defective patch from CrowdStrike, Windows VMs may crash and might not be able to reboot. Windows VMs that are currently up and running should no longer be impacted. According to CrowdStrike, 80% of Windows VMs experiencing this issue will self-heal during a reboot. Google teams are continuing to work with CrowdStrike on helping customers recover their VMs and proactively reaching out to affected customers. We will provide an update by Friday, 2024-07-19 17:00 US/Pacific with current details. If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. Diagnosis: Windows VM are crashing and going into an unexpected reboot. Impacted users may observe Serial port 1 showing the call trace, SYSTEM_THREAD_EXCEPTION_NOT_HANDLED Csagent.sys (part of the Crowdstrike Application package) 0xFFFFFFFFC0000005 0xFFFFF80E88CF033D 0xFFFF858A870FAC58 0xFFFF858A870FA4A0 Dumping stack trace: 0xFFFFF809E35317BF (pvpanic.sys+0x17BF) 0xFFFFF809E35316CB (pvpanic.sys+0x16CB) 0xFFFFF80335941B27 (ntoskrnl.exe+0x292B27) 0xFFFFF80335940AD9 (ntoskrnl.exe+0x291AD9) 0xFFFFF80335868CE7 (ntoskrnl.exe+0x1B9CE7) 0xFFFFF8033588447C (ntoskrnl.exe+0x1D547C) 0xFFFFF803358416BF (ntoskrnl.exe+0x1926BF) 0xFFFFF8033587335F (ntoskrnl.exe+0x1C435F) 0xFFFFF803356D77D0 (ntoskrnl.exe+0x0287D0) 0xFFFFF8033579D214 (ntoskrnl.exe+0x0EE214) 0xFFFFF8033587CF42 (ntoskrnl.exe+0x1CDF42) 0xFFFFF8033587893D (ntoskrnl.exe+0x1C993D) 0xFFFFF809E314033D (csagent.sys+0x0E033D) 0xFFFFF809E3115EEE (csagent.sys+0x0B5EEE) 0xFFFFF809E3117185 (csagent.sys+0x0B7185) 0xFFFFF809E334A037 (csagent.sys+0x2EA037) 0xFFFFF809E3346BB4 (csagent.sys+0x2E6BB4) 0xFFFFF809E30C68C1 (csagent.sys+0x0668C1) 0xFFFFF809E30C597E (csagent.sys+0x06597E) 0xFFFFF809E30C56EB (csagent.sys+0x0656EB) 0xFFFFF809E316883A (csagent.sys+0x10883A) 0xFFFFF809E30BDD3B (csagent.sys+0x05DD3B) 0xFFFFF809E30BDB57 (csagent.sys+0x05DB57) 0xFFFFF809E315D4D1 (csagent.sys+0x0FD4D1) 0xFFFFF803357B4A85 (ntoskrnl.exe+0x105A85) 0xFFFFF803358719FC (ntoskrnl.exe+0x1C29FC) Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted. Identifying Faulty "C-00000291*.sys" Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version. If your VMs are affected, please follow the Workaround Steps to fix the issue. Follow offline repair (Step 1 to 3) Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft. Navigate to the D:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Power down the Rescue VM detach the secondary disk Re-attach the VM to the original VM and boot

Summary: Windows VMs using CrowdStrike are crashing. Description: We are experiencing an issue with Windows VMs running CrowdStrike on Google Compute Engine. CrowdStrike has confirmed that a faulty update to the CrowdStrike Falcon agent was deployed beginning at 04:09 UTC July 19. After having automatically received a defective patch from CrowdStrike, Windows VMs may crash and might not be able to reboot. Windows VMs that are currently up and running should no longer be impacted. According to CrowdStrike, 80% of Windows VMs experiencing this issue will self-heal during a reboot. Google teams are continuing to work with CrowdStrike on helping customers recover their VMs and proactively reaching out to affected customers. We will provide an update by Friday, 2024-07-19 15:00 US/Pacific with current details. If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. Diagnosis: Windows VM are crashing and going into an unexpected reboot. Impacted users may observe Serial port 1 showing the call trace, SYSTEM_THREAD_EXCEPTION_NOT_HANDLED Csagent.sys (part of the Crowdstrike Application package) 0xFFFFFFFFC0000005 0xFFFFF80E88CF033D 0xFFFF858A870FAC58 0xFFFF858A870FA4A0 Dumping stack trace: 0xFFFFF809E35317BF (pvpanic.sys+0x17BF) 0xFFFFF809E35316CB (pvpanic.sys+0x16CB) 0xFFFFF80335941B27 (ntoskrnl.exe+0x292B27) 0xFFFFF80335940AD9 (ntoskrnl.exe+0x291AD9) 0xFFFFF80335868CE7 (ntoskrnl.exe+0x1B9CE7) 0xFFFFF8033588447C (ntoskrnl.exe+0x1D547C) 0xFFFFF803358416BF (ntoskrnl.exe+0x1926BF) 0xFFFFF8033587335F (ntoskrnl.exe+0x1C435F) 0xFFFFF803356D77D0 (ntoskrnl.exe+0x0287D0) 0xFFFFF8033579D214 (ntoskrnl.exe+0x0EE214) 0xFFFFF8033587CF42 (ntoskrnl.exe+0x1CDF42) 0xFFFFF8033587893D (ntoskrnl.exe+0x1C993D) 0xFFFFF809E314033D (csagent.sys+0x0E033D) 0xFFFFF809E3115EEE (csagent.sys+0x0B5EEE) 0xFFFFF809E3117185 (csagent.sys+0x0B7185) 0xFFFFF809E334A037 (csagent.sys+0x2EA037) 0xFFFFF809E3346BB4 (csagent.sys+0x2E6BB4) 0xFFFFF809E30C68C1 (csagent.sys+0x0668C1) 0xFFFFF809E30C597E (csagent.sys+0x06597E) 0xFFFFF809E30C56EB (csagent.sys+0x0656EB) 0xFFFFF809E316883A (csagent.sys+0x10883A) 0xFFFFF809E30BDD3B (csagent.sys+0x05DD3B) 0xFFFFF809E30BDB57 (csagent.sys+0x05DB57) 0xFFFFF809E315D4D1 (csagent.sys+0x0FD4D1) 0xFFFFF803357B4A85 (ntoskrnl.exe+0x105A85) 0xFFFFF803358719FC (ntoskrnl.exe+0x1C29FC) Workaround: Workaround Steps for individual hosts: Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then: Boot Windows into Safe Mode or the Windows Recovery Environment NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Boot the host normally. Note: Bitlocker-encrypted hosts may require a recovery key. Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted. Identifying Faulty "C-00000291*.sys" Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version. If your VMs are affected, please follow the Workaround Steps to fix the issue. Follow offline repair (Step 1 to 3) Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk **NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft. Navigate to the D:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Power down the Rescue VM detach the secondary disk Re-attach the VM to the original VM and boot

Summary: Windows VMs using CrowdStrike are crashing. Description: We are experiencing an issue with Windows VMs running CrowdStrike on Google Compute Engine. CrowdStrike has confirmed that a faulty update to the CrowdStrike Falcon agent was deployed beginning at 04:09 UTC July 19. After having automatically received a defective patch from CrowdStrike, Windows VMs may crash and might not be able to reboot. Windows VMs that are currently up and running should no longer be impacted. According to CrowdStrike, 80% of Windows VMs experiencing this issue will self-heal during a reboot. Google teams are continuing to work with CrowdStrike on recovery efforts and proactively reaching out to affected customers to mitigate We will provide an update by Friday, 2024-07-19 13:00 US/Pacific with current details. If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. Diagnosis: Windows VM are crashing and going into an unexpected reboot. Impacted users may observe Serial port 1 showing the call trace, SYSTEM_THREAD_EXCEPTION_NOT_HANDLED Csagent.sys (part of the Crowdstrike Application package) 0xFFFFFFFFC0000005 0xFFFFF80E88CF033D 0xFFFF858A870FAC58 0xFFFF858A870FA4A0 Dumping stack trace: 0xFFFFF809E35317BF (pvpanic.sys+0x17BF) 0xFFFFF809E35316CB (pvpanic.sys+0x16CB) 0xFFFFF80335941B27 (ntoskrnl.exe+0x292B27) 0xFFFFF80335940AD9 (ntoskrnl.exe+0x291AD9) 0xFFFFF80335868CE7 (ntoskrnl.exe+0x1B9CE7) 0xFFFFF8033588447C (ntoskrnl.exe+0x1D547C) 0xFFFFF803358416BF (ntoskrnl.exe+0x1926BF) 0xFFFFF8033587335F (ntoskrnl.exe+0x1C435F) 0xFFFFF803356D77D0 (ntoskrnl.exe+0x0287D0) 0xFFFFF8033579D214 (ntoskrnl.exe+0x0EE214) 0xFFFFF8033587CF42 (ntoskrnl.exe+0x1CDF42) 0xFFFFF8033587893D (ntoskrnl.exe+0x1C993D) 0xFFFFF809E314033D (csagent.sys+0x0E033D) 0xFFFFF809E3115EEE (csagent.sys+0x0B5EEE) 0xFFFFF809E3117185 (csagent.sys+0x0B7185) 0xFFFFF809E334A037 (csagent.sys+0x2EA037) 0xFFFFF809E3346BB4 (csagent.sys+0x2E6BB4) 0xFFFFF809E30C68C1 (csagent.sys+0x0668C1) 0xFFFFF809E30C597E (csagent.sys+0x06597E) 0xFFFFF809E30C56EB (csagent.sys+0x0656EB) 0xFFFFF809E316883A (csagent.sys+0x10883A) 0xFFFFF809E30BDD3B (csagent.sys+0x05DD3B) 0xFFFFF809E30BDB57 (csagent.sys+0x05DB57) 0xFFFFF809E315D4D1 (csagent.sys+0x0FD4D1) 0xFFFFF803357B4A85 (ntoskrnl.exe+0x105A85) 0xFFFFF803358719FC (ntoskrnl.exe+0x1C29FC) Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted. Identifying Faulty "C-00000291*.sys" Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version. If your VMs are affected, please follow the Workaround Steps to fix the issue. Follow offline repair (Step 1 to 3) Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk **NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft. Navigate to the D:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Power down the Rescue VM detach the secondary disk Re-attach the VM to the original VM and boot

Summary: Windows VMs using CrowdStrike are crashing. Description: We are experiencing an issue with Windows VMs running CrowdStrike on Google Compute Engine. CrowdStrike has confirmed that a faulty update to the CrowdStrike Falcon agent was deployed beginning at 04:09 UTC July 19. After having automatically received a defective patch from CrowdStrike, Windows VMs crash and might not be able to reboot. Windows VMs that are currently up and running should no longer be impacted. According to CrowdStrike, 80% of Windows VMs experiencing this issue will self-heal during a reboot. Google teams are continuing to work with CrowdStrike on recovery efforts and proactively reaching out to affected customers to mitigate. We will provide an update by Friday, 2024-07-19 11:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Windows VM are crashing and going into an unexpected reboot. Impacted users may observe Serial port 1 showing the call trace, SYSTEM_THREAD_EXCEPTION_NOT_HANDLED Csagent.sys (part of the Crowdstrike Application package) 0xFFFFFFFFC0000005 0xFFFFF80E88CF033D 0xFFFF858A870FAC58 0xFFFF858A870FA4A0 Dumping stack trace: 0xFFFFF809E35317BF (pvpanic.sys+0x17BF) 0xFFFFF809E35316CB (pvpanic.sys+0x16CB) 0xFFFFF80335941B27 (ntoskrnl.exe+0x292B27) 0xFFFFF80335940AD9 (ntoskrnl.exe+0x291AD9) 0xFFFFF80335868CE7 (ntoskrnl.exe+0x1B9CE7) 0xFFFFF8033588447C (ntoskrnl.exe+0x1D547C) 0xFFFFF803358416BF (ntoskrnl.exe+0x1926BF) 0xFFFFF8033587335F (ntoskrnl.exe+0x1C435F) 0xFFFFF803356D77D0 (ntoskrnl.exe+0x0287D0) 0xFFFFF8033579D214 (ntoskrnl.exe+0x0EE214) 0xFFFFF8033587CF42 (ntoskrnl.exe+0x1CDF42) 0xFFFFF8033587893D (ntoskrnl.exe+0x1C993D) 0xFFFFF809E314033D (csagent.sys+0x0E033D) 0xFFFFF809E3115EEE (csagent.sys+0x0B5EEE) 0xFFFFF809E3117185 (csagent.sys+0x0B7185) 0xFFFFF809E334A037 (csagent.sys+0x2EA037) 0xFFFFF809E3346BB4 (csagent.sys+0x2E6BB4) 0xFFFFF809E30C68C1 (csagent.sys+0x0668C1) 0xFFFFF809E30C597E (csagent.sys+0x06597E) 0xFFFFF809E30C56EB (csagent.sys+0x0656EB) 0xFFFFF809E316883A (csagent.sys+0x10883A) 0xFFFFF809E30BDD3B (csagent.sys+0x05DD3B) 0xFFFFF809E30BDB57 (csagent.sys+0x05DB57) 0xFFFFF809E315D4D1 (csagent.sys+0x0FD4D1) 0xFFFFF803357B4A85 (ntoskrnl.exe+0x105A85) 0xFFFFF803358719FC (ntoskrnl.exe+0x1C29FC) Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted. Identifying Faulty "C-00000291*.sys" Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version. If your VMs are affected, please follow the Workaround Steps to fix the issue. Follow offline repair (Step 1 to 3) Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk **NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft. Navigate to the D:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Power down the Rescue VM detach the secondary disk Re-attach the VM to the original VM and boot

Summary: Windows VMs using CrowdStrike are crashing. Description: We are experiencing an issue with Windows VMs running CrowdStrike on Google Compute Engine. CrowdStrike has confirmed that a faulty update to the CrowdStrike Falcon agent was deployed beginning at 04:09 UTC July 19. After having automatically received a defective patch from CrowdStrike, Windows VMs crash and might not be able to reboot. Windows VMs that are currently up and running should no longer be impacted. According to CrowdStrike, 80% of Windows VMs experiencing this issue will self-heal during a reboot. We will provide an update by Friday, 2024-07-19 09:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Windows VM are crashing and going into an unexpected reboot. Impacted users may observe Serial port 1 showing the call trace, SYSTEM_THREAD_EXCEPTION_NOT_HANDLED Csagent.sys (part of the Crowdstrike Application package) 0xFFFFFFFFC0000005 0xFFFFF80E88CF033D 0xFFFF858A870FAC58 0xFFFF858A870FA4A0 Dumping stack trace: 0xFFFFF809E35317BF (pvpanic.sys+0x17BF) 0xFFFFF809E35316CB (pvpanic.sys+0x16CB) 0xFFFFF80335941B27 (ntoskrnl.exe+0x292B27) 0xFFFFF80335940AD9 (ntoskrnl.exe+0x291AD9) 0xFFFFF80335868CE7 (ntoskrnl.exe+0x1B9CE7) 0xFFFFF8033588447C (ntoskrnl.exe+0x1D547C) 0xFFFFF803358416BF (ntoskrnl.exe+0x1926BF) 0xFFFFF8033587335F (ntoskrnl.exe+0x1C435F) 0xFFFFF803356D77D0 (ntoskrnl.exe+0x0287D0) 0xFFFFF8033579D214 (ntoskrnl.exe+0x0EE214) 0xFFFFF8033587CF42 (ntoskrnl.exe+0x1CDF42) 0xFFFFF8033587893D (ntoskrnl.exe+0x1C993D) 0xFFFFF809E314033D (csagent.sys+0x0E033D) 0xFFFFF809E3115EEE (csagent.sys+0x0B5EEE) 0xFFFFF809E3117185 (csagent.sys+0x0B7185) 0xFFFFF809E334A037 (csagent.sys+0x2EA037) 0xFFFFF809E3346BB4 (csagent.sys+0x2E6BB4) 0xFFFFF809E30C68C1 (csagent.sys+0x0668C1) 0xFFFFF809E30C597E (csagent.sys+0x06597E) 0xFFFFF809E30C56EB (csagent.sys+0x0656EB) 0xFFFFF809E316883A (csagent.sys+0x10883A) 0xFFFFF809E30BDD3B (csagent.sys+0x05DD3B) 0xFFFFF809E30BDB57 (csagent.sys+0x05DB57) 0xFFFFF809E315D4D1 (csagent.sys+0x0FD4D1) 0xFFFFF803357B4A85 (ntoskrnl.exe+0x105A85) 0xFFFFF803358719FC (ntoskrnl.exe+0x1C29FC) Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted. Identifying Faulty "C-00000291*.sys" Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version. If your VMs are affected, please follow the Workaround Steps to fix the issue. Follow offline repair (Step 1 to 3) Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk **NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft. Navigate to the D:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Power down the Rescue VM detach the secondary disk Re-attach the VM to the original VM and boot

Summary: Windows VMs using CrowdStrike are crashing. Description: We are experiencing an issue with Windows VMs running CloudStrike on Google Compute Engine. CloudStrike has confirmed that a faulty update to the CloudStrike Falcon agent was deployed beginning at 04:09 UTC July 19. After having automatically received a defective patch from CloudStrike, Windows VMs crash and might not be able to reboot. Windows VMs that are currently up and running should no longer be impacted. According to CrowdStrike, 80% of Windows VMs experiencing this issue will self-heal during a reboot. We will provide an update by Friday, 2024-07-19 09:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Windows VM are crashing and going into an unexpected reboot. Impacted users may observe Serial port 1 showing the call trace, SYSTEM_THREAD_EXCEPTION_NOT_HANDLED Csagent.sys (part of the Crowdstrike Application package) 0xFFFFFFFFC0000005 0xFFFFF80E88CF033D 0xFFFF858A870FAC58 0xFFFF858A870FA4A0 Dumping stack trace: 0xFFFFF809E35317BF (pvpanic.sys+0x17BF) 0xFFFFF809E35316CB (pvpanic.sys+0x16CB) 0xFFFFF80335941B27 (ntoskrnl.exe+0x292B27) 0xFFFFF80335940AD9 (ntoskrnl.exe+0x291AD9) 0xFFFFF80335868CE7 (ntoskrnl.exe+0x1B9CE7) 0xFFFFF8033588447C (ntoskrnl.exe+0x1D547C) 0xFFFFF803358416BF (ntoskrnl.exe+0x1926BF) 0xFFFFF8033587335F (ntoskrnl.exe+0x1C435F) 0xFFFFF803356D77D0 (ntoskrnl.exe+0x0287D0) 0xFFFFF8033579D214 (ntoskrnl.exe+0x0EE214) 0xFFFFF8033587CF42 (ntoskrnl.exe+0x1CDF42) 0xFFFFF8033587893D (ntoskrnl.exe+0x1C993D) 0xFFFFF809E314033D (csagent.sys+0x0E033D) 0xFFFFF809E3115EEE (csagent.sys+0x0B5EEE) 0xFFFFF809E3117185 (csagent.sys+0x0B7185) 0xFFFFF809E334A037 (csagent.sys+0x2EA037) 0xFFFFF809E3346BB4 (csagent.sys+0x2E6BB4) 0xFFFFF809E30C68C1 (csagent.sys+0x0668C1) 0xFFFFF809E30C597E (csagent.sys+0x06597E) 0xFFFFF809E30C56EB (csagent.sys+0x0656EB) 0xFFFFF809E316883A (csagent.sys+0x10883A) 0xFFFFF809E30BDD3B (csagent.sys+0x05DD3B) 0xFFFFF809E30BDB57 (csagent.sys+0x05DB57) 0xFFFFF809E315D4D1 (csagent.sys+0x0FD4D1) 0xFFFFF803357B4A85 (ntoskrnl.exe+0x105A85) 0xFFFFF803358719FC (ntoskrnl.exe+0x1C29FC) Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted. Identifying Faulty "C-00000291*.sys" Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version. If your VMs are affected, please follow the Workaround Steps to fix the issue. Follow offline repair (Step 1 to 3) Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk **NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft. Navigate to the D:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Power down the Rescue VM detach the secondary disk Re-attach the VM to the original VM and boot

Summary: Windows VMs using Crowdstrike’s csagent.sys are crashing and going into unexpected reboot Description: We are experiencing an issue with Windows VMs running Cloudstrike on Google Compute Engine. After having automatically received a defective patch from Cloudstrike, Windows VMs crash and will not be able to reboot. Windows VMs that are currently up and running should no longer be impacted. We will provide an update by Friday, 2024-07-19 09:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Windows VM are crashing and going into an unexpected reboot. Impacted users may observe Serial port 1 showing the call trace, SYSTEM_THREAD_EXCEPTION_NOT_HANDLED Csagent.sys (part of the Crowdstrike Application package) 0xFFFFFFFFC0000005 0xFFFFF80E88CF033D 0xFFFF858A870FAC58 0xFFFF858A870FA4A0 Dumping stack trace: 0xFFFFF809E35317BF (pvpanic.sys+0x17BF) 0xFFFFF809E35316CB (pvpanic.sys+0x16CB) 0xFFFFF80335941B27 (ntoskrnl.exe+0x292B27) 0xFFFFF80335940AD9 (ntoskrnl.exe+0x291AD9) 0xFFFFF80335868CE7 (ntoskrnl.exe+0x1B9CE7) 0xFFFFF8033588447C (ntoskrnl.exe+0x1D547C) 0xFFFFF803358416BF (ntoskrnl.exe+0x1926BF) 0xFFFFF8033587335F (ntoskrnl.exe+0x1C435F) 0xFFFFF803356D77D0 (ntoskrnl.exe+0x0287D0) 0xFFFFF8033579D214 (ntoskrnl.exe+0x0EE214) 0xFFFFF8033587CF42 (ntoskrnl.exe+0x1CDF42) 0xFFFFF8033587893D (ntoskrnl.exe+0x1C993D) 0xFFFFF809E314033D (csagent.sys+0x0E033D) 0xFFFFF809E3115EEE (csagent.sys+0x0B5EEE) 0xFFFFF809E3117185 (csagent.sys+0x0B7185) 0xFFFFF809E334A037 (csagent.sys+0x2EA037) 0xFFFFF809E3346BB4 (csagent.sys+0x2E6BB4) 0xFFFFF809E30C68C1 (csagent.sys+0x0668C1) 0xFFFFF809E30C597E (csagent.sys+0x06597E) 0xFFFFF809E30C56EB (csagent.sys+0x0656EB) 0xFFFFF809E316883A (csagent.sys+0x10883A) 0xFFFFF809E30BDD3B (csagent.sys+0x05DD3B) 0xFFFFF809E30BDB57 (csagent.sys+0x05DB57) 0xFFFFF809E315D4D1 (csagent.sys+0x0FD4D1) 0xFFFFF803357B4A85 (ntoskrnl.exe+0x105A85) 0xFFFFF803358719FC (ntoskrnl.exe+0x1C29FC) Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted. Identifying Faulty "C-00000291*.sys" Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version. If your VMs are affected, please follow the Workaround Steps to fix the issue. Follow offline repair (Step 1 to 3) Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk **NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft. Navigate to the D:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Power down the Rescue VM detach the secondary disk Re-attach the VM to the original VM and boot

Summary: Windows VMs using Crowdstrike’s csagent.sys are crashing and going into unexpected reboot Description: We are experiencing an issue with Google Compute Engine. Our engineering team continues to investigate the issue. We will provide an update by Friday, 2024-07-19 05:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Windows VM are crashing and going into an unexpected reboot. Impacted users may observe Serial port 1 showing the call trace, SYSTEM_THREAD_EXCEPTION_NOT_HANDLED Csagent.sys (part of the Crowdstrike Application package) 0xFFFFFFFFC0000005 0xFFFFF80E88CF033D 0xFFFF858A870FAC58 0xFFFF858A870FA4A0 Dumping stack trace: 0xFFFFF809E35317BF (pvpanic.sys+0x17BF) 0xFFFFF809E35316CB (pvpanic.sys+0x16CB) 0xFFFFF80335941B27 (ntoskrnl.exe+0x292B27) 0xFFFFF80335940AD9 (ntoskrnl.exe+0x291AD9) 0xFFFFF80335868CE7 (ntoskrnl.exe+0x1B9CE7) 0xFFFFF8033588447C (ntoskrnl.exe+0x1D547C) 0xFFFFF803358416BF (ntoskrnl.exe+0x1926BF) 0xFFFFF8033587335F (ntoskrnl.exe+0x1C435F) 0xFFFFF803356D77D0 (ntoskrnl.exe+0x0287D0) 0xFFFFF8033579D214 (ntoskrnl.exe+0x0EE214) 0xFFFFF8033587CF42 (ntoskrnl.exe+0x1CDF42) 0xFFFFF8033587893D (ntoskrnl.exe+0x1C993D) 0xFFFFF809E314033D (csagent.sys+0x0E033D) 0xFFFFF809E3115EEE (csagent.sys+0x0B5EEE) 0xFFFFF809E3117185 (csagent.sys+0x0B7185) 0xFFFFF809E334A037 (csagent.sys+0x2EA037) 0xFFFFF809E3346BB4 (csagent.sys+0x2E6BB4) 0xFFFFF809E30C68C1 (csagent.sys+0x0668C1) 0xFFFFF809E30C597E (csagent.sys+0x06597E) 0xFFFFF809E30C56EB (csagent.sys+0x0656EB) 0xFFFFF809E316883A (csagent.sys+0x10883A) 0xFFFFF809E30BDD3B (csagent.sys+0x05DD3B) 0xFFFFF809E30BDB57 (csagent.sys+0x05DB57) 0xFFFFF809E315D4D1 (csagent.sys+0x0FD4D1) 0xFFFFF803357B4A85 (ntoskrnl.exe+0x105A85) 0xFFFFF803358719FC (ntoskrnl.exe+0x1C29FC) Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted. Identifying Faulty "C-00000291*.sys" Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version. If your VMs are affected, please follow the Workaround Steps to fix the issue. Follow offline repair (Step 1 to 3) Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk Navigate to the D:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Power down the Rescue VM detach the secondary disk Re-attach the VM to the original VM and boot

Summary: Windows VMs using Crowdstrike’s csagent.sys are crashing and going into unexpected reboot Description: We are experiencing an issue with Google Compute Engine. Our engineering team continues to investigate the issue. We will provide an update by Friday, 2024-07-19 02:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Windows VM are crashing and going into an unexpected reboot. Impacted users may observe Serial port 1 showing the call trace, SYSTEM_THREAD_EXCEPTION_NOT_HANDLED Csagent.sys (part of the Crowdstrike Application package) 0xFFFFFFFFC0000005 0xFFFFF80E88CF033D 0xFFFF858A870FAC58 0xFFFF858A870FA4A0 Dumping stack trace: 0xFFFFF809E35317BF (pvpanic.sys+0x17BF) 0xFFFFF809E35316CB (pvpanic.sys+0x16CB) 0xFFFFF80335941B27 (ntoskrnl.exe+0x292B27) 0xFFFFF80335940AD9 (ntoskrnl.exe+0x291AD9) 0xFFFFF80335868CE7 (ntoskrnl.exe+0x1B9CE7) 0xFFFFF8033588447C (ntoskrnl.exe+0x1D547C) 0xFFFFF803358416BF (ntoskrnl.exe+0x1926BF) 0xFFFFF8033587335F (ntoskrnl.exe+0x1C435F) 0xFFFFF803356D77D0 (ntoskrnl.exe+0x0287D0) 0xFFFFF8033579D214 (ntoskrnl.exe+0x0EE214) 0xFFFFF8033587CF42 (ntoskrnl.exe+0x1CDF42) 0xFFFFF8033587893D (ntoskrnl.exe+0x1C993D) 0xFFFFF809E314033D (csagent.sys+0x0E033D) 0xFFFFF809E3115EEE (csagent.sys+0x0B5EEE) 0xFFFFF809E3117185 (csagent.sys+0x0B7185) 0xFFFFF809E334A037 (csagent.sys+0x2EA037) 0xFFFFF809E3346BB4 (csagent.sys+0x2E6BB4) 0xFFFFF809E30C68C1 (csagent.sys+0x0668C1) 0xFFFFF809E30C597E (csagent.sys+0x06597E) 0xFFFFF809E30C56EB (csagent.sys+0x0656EB) 0xFFFFF809E316883A (csagent.sys+0x10883A) 0xFFFFF809E30BDD3B (csagent.sys+0x05DD3B) 0xFFFFF809E30BDB57 (csagent.sys+0x05DB57) 0xFFFFF809E315D4D1 (csagent.sys+0x0FD4D1) 0xFFFFF803357B4A85 (ntoskrnl.exe+0x105A85) 0xFFFFF803358719FC (ntoskrnl.exe+0x1C29FC) Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted. If your VMs are affected, please follow the Workaround Steps to fix the issue. Follow offline repair (Step 1 to 3) Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk Navigate to the D:\Windows\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Power down the Rescue VM detach the secondary disk Re-attach the VM to the original VM and boot

Summary: Windows VM are crashing and going into unexpected reboot Description: We are experiencing an issue with Google Compute Engine. Our engineering team continues to investigate the issue. We will provide an update by Friday, 2024-07-19 02:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Windows VM are crashing and going into unexpected reboot. Impacted users may observe Serial port 1 showing the call trace, SYSTEM_THREAD_EXCEPTION_NOT_HANDLED Csagent.sys (part of the Crowdstrike Application package) 0xFFFFFFFFC0000005 0xFFFFF80E88CF033D 0xFFFF858A870FAC58 0xFFFF858A870FA4A0 Dumping stack trace: 0xFFFFF809E35317BF (pvpanic.sys+0x17BF) 0xFFFFF809E35316CB (pvpanic.sys+0x16CB) 0xFFFFF80335941B27 (ntoskrnl.exe+0x292B27) 0xFFFFF80335940AD9 (ntoskrnl.exe+0x291AD9) 0xFFFFF80335868CE7 (ntoskrnl.exe+0x1B9CE7) 0xFFFFF8033588447C (ntoskrnl.exe+0x1D547C) 0xFFFFF803358416BF (ntoskrnl.exe+0x1926BF) 0xFFFFF8033587335F (ntoskrnl.exe+0x1C435F) 0xFFFFF803356D77D0 (ntoskrnl.exe+0x0287D0) 0xFFFFF8033579D214 (ntoskrnl.exe+0x0EE214) 0xFFFFF8033587CF42 (ntoskrnl.exe+0x1CDF42) 0xFFFFF8033587893D (ntoskrnl.exe+0x1C993D) 0xFFFFF809E314033D (csagent.sys+0x0E033D) 0xFFFFF809E3115EEE (csagent.sys+0x0B5EEE) 0xFFFFF809E3117185 (csagent.sys+0x0B7185) 0xFFFFF809E334A037 (csagent.sys+0x2EA037) 0xFFFFF809E3346BB4 (csagent.sys+0x2E6BB4) 0xFFFFF809E30C68C1 (csagent.sys+0x0668C1) 0xFFFFF809E30C597E (csagent.sys+0x06597E) 0xFFFFF809E30C56EB (csagent.sys+0x0656EB) 0xFFFFF809E316883A (csagent.sys+0x10883A) 0xFFFFF809E30BDD3B (csagent.sys+0x05DD3B) 0xFFFFF809E30BDB57 (csagent.sys+0x05DB57) 0xFFFFF809E315D4D1 (csagent.sys+0x0FD4D1) 0xFFFFF803357B4A85 (ntoskrnl.exe+0x105A85) 0xFFFFF803358719FC (ntoskrnl.exe+0x1C29FC) Workaround: We recommend the affected users to work with the application package provider and refer to, https://supportportal.crowdstrike.com/s/article/Tech-Alert-Windows-crashes-related-to-Falcon-Sensor-2024-07-19 for additional information.

Summary: Windows VM are crashing and going into unexpected reboot Description: We are experiencing an issue with Google Compute Engine. Our engineering team continues to investigate the issue. We will provide an update by Friday, 2024-07-19 01:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Windows VM are crashing and going into unexpected reboot. Impacted users may observe Serial port 1 showing the call trace, SYSTEM_THREAD_EXCEPTION_NOT_HANDLED Csagent.sys (part of the Crowdstrike Application package) 0xFFFFFFFFC0000005 0xFFFFF80E88CF033D 0xFFFF858A870FAC58 0xFFFF858A870FA4A0 Dumping stack trace: 0xFFFFF809E35317BF (pvpanic.sys+0x17BF) 0xFFFFF809E35316CB (pvpanic.sys+0x16CB) 0xFFFFF80335941B27 (ntoskrnl.exe+0x292B27) 0xFFFFF80335940AD9 (ntoskrnl.exe+0x291AD9) 0xFFFFF80335868CE7 (ntoskrnl.exe+0x1B9CE7) 0xFFFFF8033588447C (ntoskrnl.exe+0x1D547C) 0xFFFFF803358416BF (ntoskrnl.exe+0x1926BF) 0xFFFFF8033587335F (ntoskrnl.exe+0x1C435F) 0xFFFFF803356D77D0 (ntoskrnl.exe+0x0287D0) 0xFFFFF8033579D214 (ntoskrnl.exe+0x0EE214) 0xFFFFF8033587CF42 (ntoskrnl.exe+0x1CDF42) 0xFFFFF8033587893D (ntoskrnl.exe+0x1C993D) 0xFFFFF809E314033D (csagent.sys+0x0E033D) 0xFFFFF809E3115EEE (csagent.sys+0x0B5EEE) 0xFFFFF809E3117185 (csagent.sys+0x0B7185) 0xFFFFF809E334A037 (csagent.sys+0x2EA037) 0xFFFFF809E3346BB4 (csagent.sys+0x2E6BB4) 0xFFFFF809E30C68C1 (csagent.sys+0x0668C1) 0xFFFFF809E30C597E (csagent.sys+0x06597E) 0xFFFFF809E30C56EB (csagent.sys+0x0656EB) 0xFFFFF809E316883A (csagent.sys+0x10883A) 0xFFFFF809E30BDD3B (csagent.sys+0x05DD3B) 0xFFFFF809E30BDB57 (csagent.sys+0x05DB57) 0xFFFFF809E315D4D1 (csagent.sys+0x0FD4D1) 0xFFFFF803357B4A85 (ntoskrnl.exe+0x105A85) 0xFFFFF803358719FC (ntoskrnl.exe+0x1C29FC) Workaround: We recommend the affected users to work with the application package provider.

Summary: Windows VM are crashing and going into unexpected reboot Description: We are experiencing an issue with Google Compute Engine. Our engineering team continues to investigate the issue. We will provide an update by Friday, 2024-07-19 00:15 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Windows VM are crashing and going into unexpected reboot Workaround: None at this time.