Here's a quick explanation of what is going on.
In the late hours of July 18, CrowdStrike released an update which saw Windows machines BSoD (Blue Screen of Death) across the world. Initially this was reported as a Microsoft centric issue, with Azure and Office365 being impacted, but it later transpired that CrowdStrike's update of its Falcon Sensor which detects and reacts to threats to systems, was the cause. Official confirmation of CrowdStrike being the root cause has been made, and a workaround fix has been issued. Things are slowly returning to normal, but as each time zone wakes up, more cases are being reported.
Update 05:49 PDT
Running a Windows Client / Server virtual machine on Microsoft's Azure? The suggested fix from Microsoft may see you rebooting your machine up to 15 times!
If your Windows Client or Server VM is running the CrowdStrike Falcon agent, then the BSoD bug may see your VM stuck in a restarting state. Using the Azure Portal or Azure CLI / Shell you need to reboot your VMs a number of times. Microsoft states that some users have rebooted up to 15 times to get past this issue.
Update 04:58 PDT
George Kurtz, President and CEO of Crowdstrike has been interviewed by NBC News and issued an apology for the disruption caused by the global outage triggered by CrowdStrike's update
"We're deeply sorry for the impact that we've caused to customers, to travellers, to anyone affected by this, including our companies"
Update 04:11 PDT
Need to fix the issue quickly? Here are the steps that you need to take. Note that this may not work for everyone, and you do so at your own risk. This fix comes courtesy of Brody Nisbet, CrowdStrike's Director of Threat Hunting.
1. Boot Windows into Safe Mode or WRE.
2. Go to C:\Windows\System32\drivers\CrowdStrike
3. Locate and delete file matching "C-00000291*.sys"
4. Boot normally.
Microsoft now says that the "underlying cause" of the issue has now been fixed for its apps. Users should experience a "residual impact" that should decline of the next few hours.
Update 02:51 PDT
George Kurtz, President and CEO of Crowdstrike has released a statement via X (formerly Twitter).
"CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers."
In financial news, CNBC reports that shares in CrowdStrike have fallen sharply, around 20% in U.S. premarket trading. Microsoft has also seen a 2.5% drop in premarket trading. BBC Economics Editor Faisal Islam states that in unofficial trading, CrowdStrike has lost 21% of its value, approximately $16 billion overnight. But this has yet to be confirmed.
Update 02:01 PDT
UK broadcaster BBC, are reporting that Microsoft are linking the issue to CrowdStrike's update, this is the first time that Microsoft has publicly stated this since the news broke.
“We're aware of an issue affecting Windows devices due to an update from a third-party software platform. We anticipate a resolution is forthcoming," a Microsoft spokesperson said to the BBC.
Updated Story
It seems that a recent CrowdStrike code update is bricking Windows machines across the world. The issue which occurred late in the night of July 18 is impacting companies of all scales. In the United Kingdom, the London Stock Exchange, television companies, flight operators and train companies are impacted. The dreaded Blue Screen of Death (BSoD) is appearing on Windows machines across the world. The cause is now linked to a recent CrowdStrike update which George Kurtz, President and CEO of Crowdstrike has now confirmed.
The BSoD issue is down to a misconfigured configuration issue but it does mean that users are forced to take hands on action to potentially remedy the issue. But for now we would wait for official guidance on how to remedy the issue, but later in this story we do cover one approach which is apparently working for some users.
According to the BBC News website, Microsoft released a statement which removes doubt over issues with its own services, the focus moving to CrowdStrike's services.
We spotted the start of this issue via the creator of haveibeenpwned, Troy Hunt's post on X, formerly Twitter.
Something super weird happening right now: just been called by several totally different media outlets in the last few minutes, all with Windows machines suddenly BSoD’ing (Blue Screen of Death). Anyone else seen this? Seems to be entering recovery mode: pic.twitter.com/DxdLyA9BLAJuly 19, 2024
We've been monitoring this issue and in the first few hours there was plenty of finger-pointing on social media, nothing official was released until 05:50 EDT, when Microsoft hinted that a "third-party" was to blame, and moments later the CrowdStrike statement was released.
It's confirmed !! Crowdstrike Issue Guys, they are working on it, in about maybe 45 mins things will be fix #csagent #crowdstrike #BSOD pic.twitter.com/0mkfRbUAF8July 19, 2024
The source of the issue is a content update for CrowdStrike's Falcon Sensor product, "The intelligent, lightweight CrowdStrike Falcon sensor, unlike any other, blocks attacks on your systems while capturing and recording activity as it happens to detect threats fast." according to the CrowdStrike website
What's the impact of the CrowdStrike outage?
The impact of the issue is global and it seems that today is a bad day for Windows users. CrowdStrike has confirmed that MacOS and Linux users are unaffected but airports, banks, stock exchanges, TV networks, medical services are all impacted across the world. We've compiled a list of some key areas that have been impacted during the early hours of this story.
- Reuters are reporting that IT systems for the upcoming Olympic Games in Paris are affected, with the organizers moving to a contingency process.
- United, Delta and American Airlines have issued a "global ground stop" on all of their flights. Flights already in the air will continue, and there are no apparent safety issues.
- Australian Telstra Group, a telecommunications company is also facing disruption.
- Airports across the UK are reporting delays and flight suspensions. Barcodes used for security checks at London Gatwick are not working, with security checks conducted manually.
- India's Delhi airport has resorted to manual processing of passengers and flight times communicated via a whiteboard.
- Railway companies are reporting delays.
- Sky TV and BBC Children's channel CBBC are off the air, with Sky running old stories.
Is this a hack?
Right now, there is no evidence that this is an orchestrated attack with a malicious intent. No hacker groups have come forward to claim the hack, and at the time of writing, it is believed that there are no personal data loss or safety issues.
The issue doesn't seem linked to any cyber attacks, merely a bad update is likely to blame. A bad update which has impacted many aspects of our digital lives.
What is CrowdStrike?
CrowdStrike is an American cybersecurity company. Based in Austin, Texas, Crowdstrike provides "cloud workload protection and endpoint security." The goal of the software is to prevent hacks and outages, so it seems ironic that it could now be the cause of a global IT outage. The alleged cause of the issue is CrowdStrike's Falcon Sensor, a tool that analyzes connections to and from the wider Internet for malicious behavior.
Brody Nisbet, CrowdStrike's Director of Threat Hunting has confirmed that the issue lies with CrowdStrike, but the issues lies with a "faulty channel file" and Nisbet suggests a workaround for some of those stuck in a BSOD boot loop. The fix has to be manually applied to each affected machine. Remotely managed systems can (hopefully) do this from afar, but for others will need a System Administrator (sysadmin) or IT support team member to perform the task. Remember to say thanks to your sysadmin today!
There is a faulty channel file, so not quite an update. There is a workaround...1. Boot Windows into Safe Mode or WRE.2. Go to C:\Windows\System32\drivers\CrowdStrike3. Locate and delete file matching "C-00000291*.sys"4. Boot normally.1/2July 19, 2024
There is a faulty channel file, so not quite an update. There is a workaround...
1. Boot Windows into Safe Mode or WRE.
2. Go to C:\Windows\System32\drivers\CrowdStrike
3. Locate and delete file matching "C-00000291*.sys"
4. Boot normally.
As the global outage was unfolding, we reached out to Tom Cheesewright, Applied Futurist who has worked with NASA, Google and Meta, for comment on this global issue.
"It will be interesting to find out if the two occurrences - Azure going down and the CrowdStrike issue - are connected. If not, it's an awful coincidence and one that has really compounded the chaos for Microsoft users. This is news because it's rare and we have to remember that, in spite of today's chaos. Cloud systems have proven to be a more reliable, more efficient and largely more secure way of operating. They're big news when they fail because so many people are affected. But if you aggregated the many small failures and cost of all the hardware we used to have in data centres, and the dusty servers in the corner of basements, I'm pretty sure we'd all come to the conclusion that the occasional failure is worth it."
This is an ongoing story and we will update as we get more information.
