Servers down after CrowdStrike update — How it happened and how to fix

Racks of servers inside a data center.

If you're managing servers you may need to cancel your weekend plans as a CrowdStrike update has caused servers to BSOD / boot loop. 

The incident does not appear to be a security incident or cyberattack, and only affects Windows hosts, with CrowdStrike saying Linux and Mac are not affected. 

The issue was first reported 19:00 UTC on July 18 and was acknowledged by CrowdStrike in the early hours of July 19.

"CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts," CrowdStrike CEO George Kurtz wrote on Twitter/X.

"This is not a security incident or cyberattack," he added, "the issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website."

(Image credit: CrowdStrike)

The good news is that a fix has already been found. The bad news is that as servers are not booting it is likely many will require manual intervention. CrowdStrike gave the following instructions on how to fix the issue.

1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching C-00000291*.sys* and delete it
4. Boot the host normally

Microsoft later issued further advice:

1. We recommend customers that are able to, to restore from a backup from before 19:00 UTC on the 18th of July
2. Alternatively, attempt to repair the OS disk offline.
3. Attach a disk to VM for offline repair (Encrypted disks may need further instructions)
4. Once the disk is attached delete the  Windows/System/System32/Drivers/CrowdStrike/C00000291*.sys file
5. We can confirm the affected update has been pulled by CrowdStrike. Customers that are continuing to experience issues should reach out to CrowdStrike for additional assistance.

Who is affected by the CloudStrike update?

The CrowdStrike update has affected Windows devices and Virtual Machines running Windows Client and Windows Servers running the CrowdStrike Falcon agent. Personal PCs running Windows are not affected.

It's not yet known exactly how many machines have been affected but it's already had a large impact on the globe especially in Europe with Visa, Amazon, and Microsoft all reporting issues. There have also been reports of airlines and hospitals having issues. We won't know the full extent of the impact until later in the day.

How to fix the CrowdStrike issue?

Essentially, you need to delete the file matching C-00000291*.sys

You can do that by

1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching C-00000291*.sys and delete it

or

You may need to manually remove /update the OS disk

What is CrowdStrike?

CrowdStrike is a cybersecurity company behind software used by some of the largest companies and institutions around the world, including hospitals, airports, banks, and many businesses listed in the Fortune 500.

You might also like

• We've rounded up the best managed WordPress hosting
• And here are the best Windows hosting options
• Linux vs Windows Hosting - which is ideal for you?