The U.S. Department of Homeland Security, Treasury Department and FireEye are among the most prominent victims affected by the supply chain attack on SolarWinds network monitoring software. But these data breaches are just scratching the surface of one of the most significant foreign hacking incidents in history – one that will have long-lasting repercussions.
SolarWinds estimates that between last March and June, roughly 18,000 user organizations downloaded updates of its Orion software that Russian APT actors allegedly corrupted with Sunburst backdoor malware. That attack allowed the culprits to perform reconnaissance, elevate their privileges, move laterally and steal data.
Now SolarWinds customers – over 300,000 of them, including most of the Fortune 500 – must determine whether or not they were among those impacted by the cyber espionage operation.
Near-term: Stop the bleeding
For starters, customers must confirm precisely what data and systems were affected, then mitigate the damage and remove all signs of persistence before they can safely use the Orion software again. In the longer term, companies will also have to take a hard look at new safeguards and internal security policies for all third-party software, especially programs that enable highly privileged visibility and access into sensitive systems.
In light of the attack, DHS’ Cybersecurity and Infrastructure Security Agency (CISA) ordered federal agencies to “immediately disconnect or power down SolarWinds Orion products, versions 2019.4 through 2020.2.1 HF1, from their network” and block all connections from systems using those products. Corporations may wish to do the same to prevent any further cyber espionage activity from taking place. But that’s just one facet of what should be a far more comprehensive response.
“I would be asking the team to stop and drop any other work, assess the software and versions in use, see if the malicious updates were applied, and then respond accordingly,” said Ben Johnson, former NSA hacker, O365 security expert and CTO of SaaS security firm Obsidian.
To that end, John Mancini, senior product manager at Vectra, said that a core point of the DHS’ guidance for remediating the SolarWinds hack is to analyze for any listed indicators of compromise and then “identify potential behaviors in metadata that may be related to the compromise.”
Another critical part of that response will be keeping the public informed. “In the event data or critical systems were compromised, companies should be taking the unfortunate but necessary step of public disclosure and assessing not just the damage caused by SolarWinds’ compromise, but also the factors within their own networks that contributed to attackers moving freely between systems and networks,” said Jack Mannino, CEO at nVisium.
Kelvin Coleman, executive director of the National Cyber Security Alliance, listed out several crucial steps companies must implement, including “executing any incident response plans they have through their security teams/SOC; determining what data has been explicitly compromised or stolen in the process; simultaneously contacting suppliers, vendors, partners, etc. to alert them that they’ve been breached; enacting threat hunting protocols with a zero-trust philosophy in mind to figure out if there’s any evidence of continued intrusion in their networks; updating passwords, encryption measures and MFA ‘secrets’ credentials, [and] preparing a public disclosure strategy, especially if public/consumer data is determined to have been compromised.”
Naturally, as the investigation continues, more information will surface.
“For any customer of SolarWinds Orion, it is worth digging as deep as possible to understand the implications,” added Brandon Hoffman, chief information security officer at Netenrich. "It's not clear whether this is a flaw that SolarWinds totally understands yet. If they do, a fix needs to be issued immediately. If not, it may be worth shutting down that system until there is one." (A SolarWinds advisory does cite two hot fixes that the company recommends downloading.)
Shutting down your system "may seem like overkill, but the risk is obvious, especially for targets considered higher priority," Hoffman continued. "We still don't know enough to determine if the attackers have been completely rooted out of the breached systems or even if the full extent of their lateral movements are known.”
This, said Johnson, is why "if you are impacted – or at least have the targeted software – you are going to have to do both a broad and potentially deep sweep of your environment as these actors appear sophisticated and therefore would try to embed their persistence in your environment.”
But how long will this deep sweep take? Long enough to look for any signs of persistence, while also ensuring that whatever systems do not need to rely on SolarWinds are isolated from its capabilities.
“After months of incident response, hunting, patching, and tuning monitoring systems would it be safe to reconnect again? Going forward, the SolarWinds systems should be segmented away from other parts of the environment so that the impact of any future weaknesses is mitigated," said Johnson.
Indeed, “many customers are skeptical of re-enabling this software in their environments until they have assurance that the malicious code was removed from public releases," added Mannino. "Even if the malicious code were removed from the publicly available versions of these products and the attackers were successfully removed from the environment, it will take a wait-and-see approach for many organizations to re-enable these software packages.”
Long-term: Newfound scrutiny of third-party software
Over the long term, certain companies or agencies are also likely to use this incident as a turning point to justify additional scrutiny of third-party software, and safeguards against its abuse.
For instance, the SolarWinds hack will likely lead to "stronger assessments of vendors and more defense in depth,” said Johnson. “Anything that becomes critical infrastructure and has pervasive access should be heavily monitored as not only would external adversaries be a risk, but any internal users who have access to it as well.”
As reported by Krebs on Security, a SolarWinds support advisory noted that its Orion software may not always work right unless it its file directories are exempted from antivirus scans and group policy object restrictions. For some organizations, this incident may spell the end of such exceptions.
“Internal security policies must take a trust but validate approach to all software that they deploy,” said Mancini. “Many third-party tools will trip defensive technologies, but that does not justify blanket whitelisting of these tools. An effective defensive posture must continue to keep these tools in view and to continue to monitor for new behaviors and deviations from traditional behaviors."
Meanwhile, Joe Slowik, senior security researcher at DomainTools, suggested that organizations may want to consider investing in security solutions designed to monitor network communications for anomalous traffic flows, “such as a SolarWinds server attempting to resolve a new, unexpected domain,” which might suggest your systems are receiving instructions from an attacker. “Thorough understanding of our own networks and visibility into network traffic flows can defeat even the most complex adversaries,” Slowik explained.
Of course, rarely do security professionals encounter APT operations quite as sophisticated as this one. As FireEye noted in its own report on the attack, Sunburst malware “masquerades its network traffic as the Orion Improvement Program (OIP) protocol and stores reconnaissance results within legitimate plugin configuration files, allowing it to blend in with legitimate SolarWinds activity.” This is one of multiple stealth capabilities that helped the operation go undetected for so long, along with a two-week dormancy period and the use of “obfuscated blocklists to identify forensic and anti-virus tools running as processes, services, and drivers.”
Indeed, Matt Ashburn, head of strategic initiatives and chief information security officer at the National Security Council, said that effective detection and mitigation of such supply chain threats “require concerted coordination among traditionally disparate teams, including procurement, logistics, compliance, and security teams.”
Ashburn said that organizations looking to reduce the risk of similar incidents in the future must work to “fully understand and inventory all devices -- including make, model, and supplier information, including manufacturers, resellers, and sub-suppliers” and also “research each level of the supply chain to understand supplier relationships, security practices, and analyze potential risk.”
Additionally, he recommends adopting a modern, zero-trust security architecture – perhaps one that prevents any outbound web communications “except those known and verified to be trusted connections.”
Moreover, “further segmentation of networks and consolidation of technologies to reduce the complexity of systems would also help defenders have a more focused approach," said Johnson.
“Supply chain security will be a front and center issue for many organizations as the fallout from this incident unfolds,” concluded Mannino. “In addition to traditional software security testing techniques such as code reviews and penetration testing, an increasing number of organizations may be interested in understanding how software behaves through malicious code reviews. These types of tests explore the likelihood that software contains embedded malware, through malicious code commits or by compromised third-party dependencies.”
Coleman said that moving forward, organizations are going to have to hold third-party software providers more accountable for their security. "Although this should have been status quo from the start, this incident should be a wake-up call to companies to keep security standards top of mind when vetting new third-party partners and reassessing existing ones," he said. "Contracts should stipulate regular network testing protocols and ‘right to audit’ clauses, incident response measures should be transparent, and third-party vendors should have a track record of adhering to compliance standards (e.g. HIPAA, ITAR, PCI-DSS) and abiding by industry frameworks (e.g. as outlined by NIST)."
"And while there are countless more behaviors and safeguards that businesses should be taking, it’s clear that this attack just opened up tons of eyes to the sort of destruction a supply chain attack can have," Coleman continued. "Chances are we’ll see these sorts of measures become more commonplace as companies deal with the fallout."
