A year ago, when the Log4Shell vulnerability was first disclosed, perhaps no sector responded as quickly and decisively as the federal government.
Within days of the bug’s disclosure, the Cybersecurity and Infrastructure Security Agency at the Department of Homeland Security issued an emergency directive ordering civilian federal agencies to identify all software solution stacks accepting data input from the internet, map them to a government-run GitHub repository of known software assets using the vulnerable code, patch known affected instances and request additional scrutiny for internet connected solutions that were not on the list. Agencies were given less than week to patch their affected systems or pull them from the internet.
On top of all that, CISA led a frenetic national coordination effort to remediate the bug, pulling in cybersecurity experts, members of industry, major software and hardware providers and other stakeholders to spread awareness, identify affected assets and reduce the nation’s overall attack surface by patching as many systems as possible.
If any organization was going to be able to quickly find and inoculate themselves from Log4j, it should have been the federal government.
And yet, three months later in February 2022, hackers linked to the Iranian government were able to use the vulnerability to break into an unpatched VMWare Horizon server, and federal officials did not even discover the compromise or respond to it until June 2022, according to a joint advisory by CISA and the FBI.
From the time it was introduced, researchers warned that Log4Shell would remain in the digital ecosystem for years. The Iranian incident underscores just how difficult it has been even for motivated and well-resourced organizations to stay on top of the threat and remove it from their networks.
A deep infestation
In a statement sent to SC Media, CISA executive director Eric Goldstein did not directly answer questions about why the compromise was not found until months after agencies were ordered to scour their internal software but noted that discovery efforts would be an ongoing effort and that “while organizations across government and the private sector acted with urgency to mitigate assets running vulnerable versions of Log4j, we know that malicious cyber actors moved quickly to exploit vulnerable assets and continue to do so.”
SC Media spoke with security researchers and vendors who have been tracking similar failures across the private sector, with alarming numbers of known internet-connected assets still vulnerable to Log4j today.
According to Sonatype, the software supply-chain firm that runs Apache Maven Central (the largest Java package repository), one out of every four Log4j instances downloaded from the repository are still vulnerable today. More disturbingly, researchers from Tenable found that 72% of organizations worldwide remain vulnerable to the Log4Shell, highlighting its sticking power and remediation challenges.
“These earth-shattering data is a lesson to the security community, and we clearly have a long way to go,” Brian Fox, cofounder and chief technology officer at Sonatype, said in an interview.
Log4Shell is hard to eradicate due to the sheer size of its attack surface and how easy it is to exploit.
Bugs in an operating system or a popular browser typically only affect the users of that system or browser, and the problem will be fixed once the security team patches the issue. Log4Shell is different. By targeting the logging systems that countless developers have relied on, it has a much larger attack surface and compromised millions of vulnerable Java applications and services worldwide, including Cloudflare, iCloud, Minecraft servers, Steam, TencentQQ, and Twitter.
In addition, attackers can easily exploit the vulnerability as it does not require any privileged access or special configuration. All they have to do is to locate an input field that gets logged and write a simple string of code, said Yotam Perkal, director of vulnerability research at Rezilion.
Layers of insecurity
In response to the ubiquitous nature of Log4Shell, security teams in the private sector have spent days and nights to detect and patch vulnerable versions in a timely manner. However, hunting Log4Shell is easier said than done.
Software projects are usually built on top of a mountain of dependencies where developers incorporate external software libraries into the projects and add additional functionality. While some developers knowingly and directly use log4j in their software, others may have code that relies on other open source programs in order function – and those programs could also be vulnerable. This weakness, known as transitive dependency, means undocumented pieces of code may bury the vulnerable versions of Log4Shell multiple levels deep in data, said Stephen Magill, vice president of product innovation at Sonatype.
According to Magill, approximately 70,000 open-source projects use log4j as a direct dependency, while nearly 174,000 projects use it as a transitive dependency.
“70,000 is already a substantial number showing that many projects are directly vulnerable. But if you add on transitive dependency, the number of affected projects is more than doubled, making hunting and remediation even more difficult,” Magill said.
To fully detect the vulnerable versions in all dependencies, organizations need to be more patient with running their test, said Jason Kent, hacker in residence at Cequence Security.
“We found that a lot of organizations do not wait long enough [with their tests]. They only wait for five minutes after they send the test in, and then they move on to the next test. The vulnerable versions can be deep in multiple layers, and we have seen 24 or 48 hours before something triggers and sends it back to us,” Kent explained.
However, Kent said that even if organizations fully detect the vulnerable versions and patch them quickly, they still should not drop their guard as Log4Shell can be reintroduced to the applications anytime developers add new systems or assets to their environments.
Indeed, recent research by Tenable shows that among the vulnerable assets identified during the initial disclosure, nearly one-third (29%) of them had recurrences of Log4Shell after full remediation was achieved.
“The addition of new systems and assets is the most frequent action that inadvertently reintroduces Log4Shell. If organizations do not address the issue on the left side — in the build pipeline — they will continue to deploy vulnerable code,” said Bob Huber, chief security officer and head of research at Tenable. “As it stands, many organizations address the issue on the right side — in their runtime environment — only to be replaced with another insecure build. Identifying every instance of insecure code in use is nontrivial for most organizations, including third parties.”
Detection, remediation come with a high price tag
Besides the Iranian government-sponsored attack disclosed last month, multiple other nation-state-backed actors have been reported using Log4Shell to compromise target networks. In March, for example, Chinese threat actor Deep Panda has been observed to use an identical vector to deploy a backdoor and attack mainly US-based government entities, while the North Korea’s Lazarus group has been found using the same technique to conduct cyber espionage and ransomware attacks.
Despite such reports, the number of publicly disclosed incidents targeting Log4j is surprisingly low, especially compared to other software supply chain vulnerabilities, like the Microsoft Exchange Server vulnerabilities and usage of ProxyLogon and ProxyShell. Fox argued that this number fails to represent the actual impact of Log4Shell.
“[Just because] people don’t see headlines does not mean the attacks are not happening,” Fox said.
When it comes to evaluating the impact of Log4Shell, organizations should not only look at the number of publicly disclosed attacks but also consider the economic cost of detection and remediation, said Dan Lorenc, founder and CEO of Chainguard.
“Hunting and patching Log4Shell is very costly. Over the past year, it has cost companies billions of dollars to get through the mitigations, along with a lot of security folks sacrificing their weekends and holiday breaks to handle the issues,” Lorenc said.
To quantify the cost and draw a more comprehensive picture of Log4Shell, Arctic Wolf noted that the average cost of a Log4j incident response over the past year has amounted to $90,000, while GuidePoint Security suggested that the cost for a single Log4j hunt can also reach to $33,000.
“[The high cost] was due to numerous factors, including the breadth of Log4j across various software and solutions, as well as the client’s inability to easily identify where it was running, whether it was vulnerable, and even potentially exploited. Therefore, we seem to receive more requests for assistance in helping clients perform these tasks, which obviously has supplemental costs to their typical vulnerability management processes,” said Mark Lance, vice president of DFIR and Threat Intelligence at GuidePoint Security.
“Also, given that the vulnerability took almost the whole IT department, security teams, and application owners, and even many non-security-oriented teams’ effort in the first month of its disclosure, I would estimate the mitigation of Log4j took up 10% to 12% budget from each team (including incident response) over the past year,” Alex Kozodoy, cyber research manager at Deep Instinct, added.
Moving forward
Security researchers said that organizations should take those losses as lessons that could help them better balance the prevention and mitigation of vulnerabilities in the upcoming year.
“I figured all organizations knew about their network vulnerabilities. But what I have learned [from Log4Shell] is that is not the case. Most organizations, no matter how large or small they are, are truly blind to what they have exposed to the internet,” said Justin Fier, VP of Tactical Risk and Response at Darktrace.
While some researchers suggested that on a positive note, the fallout from Log4j is helping industry and government to push transparency tools like the software bill of materials (SBOMs), Fox warned that organizations need to truly understand their software components instead of producing it just to fulfill government requirements. If not done carefully, organizations could find themselves following the same familiar pattern of panic and persistent exposure the next time a damaging flaw in a frequently used software dependency emerges.
“Be sure to get your organizational bill of materials under control. If I told you about a new vulnerability right now, and you cannot answer me if you are using that component anywhere in your portfolio, you should better get started because the next Log4Shell may be right around the corner,” Fox said.
