Forensic investigators have discovered a novel malware program used in the SolarWinds supply-chain attack – one designed specifically to seek out developers’ builds of the SolarWinds Orion IT management platform and then replace a source file with the Sunburst backdoor.
Targeting build servers in such a manner is a devious strategy, because such machines prioritize efficiency of developer use over the kind of in-depth security that’s needed to reliably detect malicious activity. SolarWinds noted this week in a new blog post that its software development and build process “is common throughout the software industry” – a troublesome notion that raises the specter of other developer environments being targeted in a similar fashion following the resounding success of this attack.
For that reason, SolarWinds and other cybersecurity experts are stressing the importance of developer organizations understanding the true nature of the threat.
SolarWinds also revealed two potentially missed opportunities to detect the supply chain attack sooner, acknowledging a pair of customer support inquiries that, in hindsight, appear to have been related to the attack campaign.
Introducing Sunspot malware
Dubbed Sunspot, the newly discovered malware spies on compromised servers in order to seek out instances of MsBuild.exe, a process that corresponds to Microsoft Visual Studio, a program used to compile Orion software builds. If the malware determines that there is an Orion build in progress, it replaces the source file “InventoryManager.cs” with the Sunburst backdoor code, according to a new Crowdstrike blog post released in conjunction with the latest SolarWinds disclosure.
Crowdstrike is one of the firms helping to investigate the attack, along with KPMG. The cybersescurity company refers to the attack operation as “StellarParticle,” and notes that the Sunspot malware was apparently built in February 2020, which fits the timeline of the campaign.
“The design of Sunspot suggests StellarParticle developers invested a lot of effort to ensure the code was properly inserted and remained undetected, and prioritized operational security to avoid revealing their presence in the build environment to SolarWinds developers,” says the blog post from the Crowdstrike Intelligence Team.
The novelty of the malware “stems from how well it blends into the build process,” said Brian Coulson, principal threat research engineer at LogRhythm. “The adversary appears to have had great knowledge of the build process prior to the execution of the attack. This likely implies that the adversary had compromised the environment some time ago and was able to gather intelligence along the way to plan and execute their attack.”
Oliver Tavakoli, CTO at Vectra, agreed. “Much reconnaissance was required to understand the lay of the land within SolarWinds and to deconstruct the build procedures for Orion,” he said. “A clear understanding of all the source files which made up Orion had to be established. An initial test of the ability to inject code into the platform was undertaken without exposing the eventual backdoor. Only once all of succeeded, was the core part of the attack carried out… It required great trade raft, lots of effort, attention to detail and supreme patience to pull off. “
In fact, Sunspot’s authors – widely assumed to be Russian state-sponsored actors – even incorporated certain checks into the malware to ensure that infected Orion builds would not result in errors that could trigger developers’ suspicions.
“The attackers clearly understood software build methodologies and build team cultures and it was interesting to see them take special precautions not to have the build of the software fail as a result of their replacement of a source file,” said Oliver Tavakoli, CTO at Vectra. “Such a build failure invariably causes a build engineer to be tasked with root-causing the failure.”
Adam Meyers, senior vice president of intelligence at Crowdstrike, told SC Media that while Sunspot is “specifically instrumented for the SolarWinds environment,” it’s also flexible enough that adversaries could “change it very easily” for future attacks. “In fact, the builders of it could use it for any number of compilers or targeted projects in another environment.”
Development environments are uniquely prone to attacks, explained Meyers, because “developers have access and privileges that average users wouldn't have” in order to do their job properly. “Build servers, build environments are optimized for build processes, so oftentimes they don't have security tools enabled on them because that might slow down or impact the build process negatively.”
Fortunately, there are now at least YARA rules and indicators of compromise that can help organizations look out for and detect this specific StellarParticle threat. Crowdstrike hopes this information makes “developers and DevOps people really think about how they monitor and secure, and what can they look for in their own build environment.”
“Learning the details of any breach will benefit attackers and defenders alike,” said Coulson. “Attackers should realize that since this is now a known attack, it’s in their best interest to stop using it, and defenders should be able to learn how to prevent and detect the known attack… So if an adversary does choose to emulate the known attack methods in the future, defenders should easily be able to detect and stop them.”
With that said, development environments have to be ready for whatever the next stealthy online assault is… which could from anywhere.
“What I think we will see is more attacks against development systems. Those have always been juicy targets but now somebody has proven just how juicy they can be,” said Brandon Hoffman, CISO at Netenrich.
Hoffman said it’s “entirely possible” that we could see future attacks that borrow tactics used in the SolarWinds incident; however, “the challenge is having source code access.”
“Access like that is exceedingly rare, and while cybercriminals could potentially gain that access, it does require sophisticated tech and also a long-term, patient plan. Those are not really hallmarks for cybercriminals. Even though they do have sophisticated code and plan well, most of the cybercriminal activity feels more like smash and grab as opposed to this activity with is more like a vault heist, although more complex.”
Much like Hoffman, Tavakoli doesn’t anticipate that the SolarWinds attack will “revolutionize” future attacks on the part of the broader cybercriminal community, he does think that certain “low-level aspects of the attack will likely get some adoption.”
On the other hand, Dirk Schrader, global vice president at New Net Technologies, isn’t about to underestimate the ambitions of malicous actors: “Copying this attack method won’t happen overnight, but it surely will,” he said.
“Looking at this attack method, the group behind the attack established a way to leap forward in the well-known cyber kill chain, bypassing at least the first three steps,” Schrader continued. “Their only task was to wait for the infected installations to ‘call home…’ All that was needed was already inside the targets, delivered via a clever way. Other groups will certainly mimic this approach, and software vendors as well as customers have to find ways to define what can be considered as ‘clean source.’”
“Others could easily pull this off,” said Meyers, noting that supply chain attacks “are the things that really cause me to lose the most sleep.”
“The x-factor is that they would need to understand the dev environment – all dev environments are not created equal. They don't all have the same tools or setups, or things of that nature. So I think that an adversary that wants to replicate this attack would still need to understand the target environment very well in order to leverage this in an attack.”
Missed opportunities for detection?
In the SolarWinds blog post, new president and CEO Sudhakar Ramakrishna noted that, after reviewing historical customer support inquiries, the company identified two previous incidents that, in hindsight, appears to be related to the Sunburst malware attack.
“We investigated the first [inquiry] in conjunction with our customer and two third-party security companies. At that time, we did not determine the root cause of the suspicious activity or identify the presence of the Sunburst malicious code within our Orion Platform software,” wrote Ramakrishna “The second incident occurred in November, and similarly, we did not identify the presence of the Sunburst malicious code.”
Full details of these two incidents weren’t shared, but it’s reasonable to ask if these two inquires were perhaps missed opportunities to short-circuit the attack, which affected 18,000 victim organizations, including many U.S. federal agencies.
Some experts said connecting the dots when an incident is reported isn’t as easy as you might think.
“My personal feeling is that these don’t feel like missed opportunities; rather, the investigation at the time wasn’t able to connect enough evidence that would lead them to detect the compromise or attack,” said Coulson. “This is common in companies where analysts aren’t able to piece together the whole picture quickly due to gaps in detections.”
Hindsight is always 20/20,” said Tavakoli. “Software stacks are incredibly complex and the OS and other software also installed on the systems housing Orion likely vary greatly from customer to customer – as are the networks which Orion is intended to monitor. Thus, as unsatisfying as it sounds, it is not that unusual for intermittent problems to occur with such systems and for the attempt to find a root cause to fail. For a better perspective, it would be interesting to know how many total support incidents which also ended inconclusively SolarWinds handled during the investigated time period.”
Futhermore, SolarWinds likely was confronted with hundreds of customer inquiries during the attack’s timespan, Tavakoli continued. “Surmising a pattern from two data points amidst this data would be difficult. Maybe if the two incidents exhibited very similar characteristics and someone within the SolarWinds support organization connected the dots and someone explicitly was looking for potential supply chain attacks the epiphany might have occurred. But that’s an abundance of low likelihood combinations.”
In his company’s latest disclosure, Ramakrishna also laid out an updated timeline of the SolarWinds attack. According to the blog post, the earliest suspicious activity dates back to September 2019, which preceded the attackers making modifications to the October 2019 version of the Orion Platform release. The culprits updated Sunspot in February 2020, and then removed Sunburst from the SolarWinds environment in June 2020.
“During that time, through to today, SolarWinds investigated various vulnerabilities in its Orion Platform,” the blog post said. “It remediated or initiated the process of remediating vulnerabilities, a regular process that continues today. However, until December 2020, the company did not identify any vulnerabilities as what we now know as Sunburst.” SolarWinds was notified of the attack on Dec. 12, 2020.
Others were not quite as forgiving. “If the suspicious activity had any relation to the Sunburst malware activity it seems this should’ve been detected in these two cases,” said Hoffman. However, “It’s very hard to speculate from outside what they saw and how far they investigated to make a clear determination.”
Schrader said that in response to this incident, some software companies may want to change procedures when a customer inquiry takes place. “As a consequence of this, customers will be alerted about any sort of weird behavior shown by a piece of software and vendors will have to ramp up their support teams with security-savvy staff to be able to properly respond and investigate,” he said.
Ramakrishna said is openly sharing details from the attack to “help the industry guard against similar attacks in the future and create safer environments for customers.”
“This is something that's never really been documented before,” Meyers said in praise of SolarWinds’ transparency. “If you look at any of the supply chain attacks that have occurred to date, nobody's ever come clean about what happened. So I think this helps defense teams and helps developers and DevOps people really think about the problem.”
