A massive IT outage has caused multiple services to be taken offline, while planes and trains were grounded as a result.
Potentially linked to an update from CrowdStrike to Microsoft users, the issue is apparently due to a misconfigured update which is causing users globally to hit a "blue screen of death" (BSOD).
Related: Security pros brace for manual system-by-system fix to CrowdStrike outage
Related: What the CrowdStrike update outage means for cybersecurity
Related: Seven tips that offer short-term and long-term fixes following the CrowdStrike outage
In a Reddit update, a post — apparently by a member of the CrowdStrike team — said they are aware of "widespread reports of BSODs on Windows hosts, occurring on multiple sensor versions. Investigating cause."
Defect in single content update
CrowdStrike CEO George Kurtz issued a statement saying the company is actively working with impacted customers, and the issue was caused by a defect found in a single content update for Windows hosts.
George Kurtz, President & CEO CrowdStrike on X
.
Confirming the outage was not “a security incident or cyberattack,” Kurtz said: "The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.
“We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers."
Reports of crashes
The disruptive file was identified as an update for CrowdStrike's Falcon sensor, which was released on July 9. A workaround was published, with CrowdStrike saying it is "aware of reports of crashes on Windows hosts related to the Falcon sensor."
CrowdStrike has confirmed that it is no longer pushing the update, “so you only have to fix the machines that were already stuck in a BSOD loop: anything that isn't impacted now shouldn't be impacted.”
Writing on X, CrowdStrike's chief threat hunter Brody Nisbet offered a workaround, recommending users:
1. Boot Windows into Safe Mode or WRE.
2. Go to C:WindowsSystem32driversCrowdStrike
3. Locate and delete file matching "C-00000291*.sys"
4. Boot normally.
Cybersecurity researcher Kevin Beaumont wrote on X that the global IT outage "is CrowdStrike as cause, not Microsoft" and that two different outages were linked together. While the Microsoft one was solved a while ago, it may be the CrowdStrike update that is causing the issue.
EXCLUSIVE: CrowdStrike founder and CEO @George_Kurtz speaks on TODAY about the major computer outages worldwide that started earlier today: “We’re deeply sorry for the impact that we’ve caused to customers, to travelers, to anyone affected by this.” pic.twitter.com/fWz6KhgrcZ
— TODAY (@TODAYshow) July 19, 2024
Infinite loop
Ilkka Turunen, field CTO at Sonatype said in an email to SC UK that the update caused a BSOD loop on any Windows machine, essentially making it boot and crash on an infinite loop.
“Making it worse is the fact that there are a significant number of Windows machines that the update was auto-installed on overnight,” he said. “There are workarounds that customers of theirs will apply, but it seems to be very manual.
"It’s definitely a supply chain style incident — what it shows is that one popular vendor botching an update can have a huge impact on its customers and how far a single well-orchestrated update can spread in a single night."
SC UK has contacted Microsoft for comment.
(This is a developing story and we will be updating this article shortly with the latest developments.)