CrowdStrike discloses new technical details behind outage

CrowdStrike said Saturday a bad “sensor configuration update” in its Falcon cybersecurity platform was to blame for a massive global computer outage. The disastrous patch knocked approximately 8.5 million Windows devices offline paralyzing airlines, hospitals and financial institutions globally.

Mac and Linux systems were not impacted and Microsoft reported Saturday that many systems been restored. 

"We currently estimate that CrowdStrike’s update affected 8.5 million Windows devices, or less than one percent of all Windows machines. While the percentage was small, the broad economic and societal impacts reflect the use of CrowdStrike by enterprises that run many critical services," wrote David Weston, Microsoft VP, Enterprise and OS Security in a blog post Saturday.

For its' part, CrowdStrike released the most complete picture to date of the technical breakdown that lead to the outage on Saturday.

“On July 19, 2024 at 04:09 UTC, as part of ongoing operations, CrowdStrike released a sensor configuration update to Windows systems. Sensor configuration updates are an ongoing part of the protection mechanisms of the Falcon platform,” CrowdStrike explained in a blog post.

It explained the “configuration update triggered a logic error resulting in a system crash and blue screen (BSOD) on impacted systems.”

In computer parlance, a logic error, sometimes called a semantic error, is a bug found in a program's source code that can trigger abnormal application behavior or system crashes. BSoD is shorthand for blue screen of death, a term used to describe the blue screen displayed indicating a system crash on a Windows computer system.

The CrowdStrike Falcon is a breach-prevention platform that uses a “unified set of cloud-delivered technologies that prevent all types of attacks — including malware and much more,” according to marketing material for CrowdStrike Falcon. It has several core functions that include antivirus, endpoint detection and response (EDR), cyber threat intelligence, managed threat hunting abilities and security hygiene. Falcon is described as a “lightweight sensor that is cloud-managed and delivered.”

CrowdStrike apology and pledge

CrowdStrike’s Saturday blog echoed previous apologetic sentiments while expressing empathy to impacted customers and a promise to do better moving forward.    

“We understand how this issue occurred and we are doing a thorough root cause analysis to determine how this logic flaw occurred. This effort will be ongoing. We are committed to identifying any foundational or workflow improvements that we can make to strengthen our process. We will update our findings in the root cause analysis as the investigation progresses.”

An apology was issued on Friday by George Kurtz, CrowdStrike’s founder and CEO, via a blog post stating, “I want to sincerely apologize directly to all of you for today’s outage.”

On Friday, the company had issued a fix for impacted systems and stressed the outage was not tied to a cybersecurity event or attack. However, security experts said the disruption of services and rush to investigate and fix systems did open the door for threat actors to take advantage of opportunity.

Full coverage of CrowdStrike's boggled update and Microsoft outage aftermath

Technical update and root cause analysis

CrowdStrike identified vulnerable systems as those running the Falcon sensor for Windows version 7.11 and above. Customers running that version of the sensor “that were online between Friday, July 19, 2024 04:09 UTC and Friday, July 19, 2024 05:27 UTC, may be impacted.”

The blog stated:

“Systems running Falcon sensor for Windows 7.11 and above that downloaded the updated configuration from 04:09 UTC to 05:27 UTC – were susceptible to a system crash.”

CrowdStrike explained that the configuration file update for the Falcon sensor are called “channel files” and tied to the platform sensor’s “behavioral protection mechanisms.”

According to CrowdStrike, updates to the sensor occur several times a day and include information on adversarial tactics, techniques, and procedures discovered by CrowdStrike. 

Microsoft also posted an update on Saturday outlining mitigation efforts and technical updates tied to the CrowdStrike incident. "Although this was not a Microsoft incident, given it impacts our ecosystem, we want to provide an update on the steps we’ve taken with CrowdStrike and others to remediate and support our customers," it wrote.

Getting into the Channel File 291 weeds

The technical specifics offered by CrowdStrike on Saturday included the Windows directory path for impacted Channel Files (C:WindowsSystem32driversCrowdStrike) and tips on finding the impacted code residing in a “file name that starts with “C-” (Channel File 291).

“The update that occurred at 04:09 UTC was designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks. The configuration update triggered a logic error that resulted in an operating system crash,” wrote CrowdStrike.

A C2 (Command and Control) framework is defined by StationX as infrastructure hackers use to control compromised systems remotely. “It consists of a C2 server, a C2 client, and a C2 agent (implant). Attackers will connect to a remote C2 server using a C2 client on their local machine.”

Mitigation includes updating Channel File 291, CrowdStrike said. It added that no updates to file will be deployed. “Falcon is still evaluating and protecting against the abuse of named pipes,” it said.

CrowdStrike remediation efforts can be found at its’ CrowdStrike Tech Alert support page. The page lists a number of resources from the recommended fix, to work around mitigation options, tips to  identify issues, recommended actions and a tool to “query to identify impacted hosts via Advanced event search”.