What you need to know
- Microsoft services across networking and cloud computing reported outages in the early hours of Friday, July 19, affecting global airlines, TV broadcasters, retail spaces, 911 operators, and more.
- Engineers of security software CrowdStrike have issued a statement acknowledging the issue and have reportedly issued a fix.
- Microsoft also took action to repair its Azure servers and remedied a separate problem for global Windows users.
In what will go down as one of the worst IT outages in history, and probably Microsoft's least favorite Friday in recent memory, the dreaded 'Blue Screen of Death' (BSOD) error screen suddenly caused worldwide outages across several major airlines, global TV news channels, banks, and even 911 emergency operator systems.
A popular website dedicated to tracking server issues, Downdetector, first reported severe problems across all of Microsoft's cloud computing services, including its primarily commercial Azure servers, subsequently affecting any infrastructure that relied upon them. Commercial flights were grounded with a "global ground stop" as hundreds of airport information screens mostly showed only the infamous Windows BSOD error.
At the same time, major cellular networks across the United States were suddenly unresponsive. TV channels saw broadcasts interrupted, affecting Sky News in the United Kingdom and several other channels in Australia. Microsoft claimed it took "mitigation actions" with investigations into its systems starting at the initial signs of issues at 6 PM ET on Thursday, July 18. Microsoft saw its services restored a few hours later, though the BSOD errors on Windows devices around the world curiously persisted.
A common link between all affected devices is reliance on security software from cybersecurity firm CrowdStrike, as its engineers became aware of an issue relating to its 'Falcon Sensor.' The specifics of what caused the fatal bug appear to be linked to a faulty kernel driver deployed by CrowdStrike known as 'csagent.sys', which causes failed boots, preventing its users from opening any installed software.
We're aware of an issue with Windows 365 Cloud PCs caused by a recent update to CrowdStrike Falcon Sensor software. This is being communicated under WP821561 in the admin center. (Cont...)July 19, 2024
Microsoft highlighted the issue and affirmed that it would soon resolve its cloud services, but CrowdStrike is ultimately responsible for fixing the BSOD problem. Initially securing endpoints and cloud workloads from potential attacks and other data breaches, CloudStrike inadvertently caused widespread outages by pushing the faulty kernel driver in its latest update.
CrowdStrike's CEO, George Kurtz, issued the following statement:
"CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed."
"We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organisations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilised to ensure the security and stability of CrowdStrike customers."
Unfortunately, CrowdStrike did not include an automatic fix for affected Windows machines released with the statement. IT admins and individual users need to manually boot Windows into safe mode and remove the faulty driver, which will likely take a while for data centers and servers across the globe impacted by this faulty software update to synchronize with the fix.
Earlier this morning, a technical issue with a vendor impacted multiple carriers, including American. As of 5:00 a.m. ET, we have been able to safely re-establish our operation. We apologize to our customers for the inconvenience.July 19, 2024
This morning's Microsoft 365 services outages appear entirely separate from the global CrowdStrike issue. Experts are calling it the most significant IT outage the world has ever seen, as infrastructure and thousands of services were knocked offline by the Windows machines that suffered a BSOD error after the CrowdStrike update.
Supermarkets, banks, hospitals, and more major institutions suffered major outages for several hours, some of which continue. A major airline in the United Kingdom, EasyJet, advised customers to arrive with three hours to spare as it had no choice but to revert to traditional pens and paper for processing passengers.
Delta Air Lines paused its global flight schedule and is "working to issue a travel waiver" to help travelers with affected flights. Simultaneously, American Airlines claims to have "re-established its operation." In Washington, D.C., the previously affected Metro system is back up and running all services as scheduled. While regular service slowly returns worldwide, shares for Crowdstrike Holdings Inc. dropped as far as 21% on the NASDAQ stock market and currently rests at -15%.
How to fix the CrowdStrike BSOD on Windows
If you have a machine that is impacted by the CrowdStrike update, you can uninstall the faulty driver and restore functionality by following the instructions below:
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file matching “C-00000291*.sys”, and delete it.
- Boot the host normally.
What exactly is a Blue Screen of Death (BSOD)?
In the context of Windows PCs, laptops, and other devices running the operating system, "Blue Screen of Death" refers to a full-screen error that signifies a critical system failure related to a specific operation, which inevitably leads to a total device crash and forced reboot.
Most of us have likely experienced the dreaded BSOD with a cryptic error message, that almost always requires research to fix. Anyone can encounter the error during an upgrade, startup, or even during normal use.
What is CrowdStrike?
CrowdStrike is a cybersecurity company based in the US. Its main focus is helping companies manage their Internet security, including protecting them from data breaches and sophisticated attacks deployed by hackers.
The cybersecurity firm offers a wide range of products, including CrowdStrike Falcon. It can be leveraged to get "real-time" indicators of attack, giving companies the upper hand by providing ample time to implement elaborate security measures.
CrowdStrike uses cloud-based AI and machine learning to detect and prevent cyberattacks. Its server crashed earlier today and is currently believed to be the root cause of the reported outage across Microsoft products.
