In testimony before the U.S. House Oversight and Homeland Security committees last week, SolarWinds’s former and current CEOs blamed an intern for creating a weak FTP server password and leaking it on GitHub – an act which may or may not have contributed to a supply chain hack that impacted users of the tech firm’s Orion IT performance monitoring platform.
But infosec thought leaders say that blaming an intern ignores the true roots of the problem, including insufficient credentials policies and access management practices – as evidenced in part by the simplicity of the password itself: “solarwinds123”.
“In placing blame on an intern for setting a production password in 2017… Solarwinds revealed deep, systemic cybersecurity failures at many levels of the organization,” said Marc Rogers, executive director of cybersecurity at Okta. “That intern’s ability to set a password of ‘solarwinds123’ on a critical production system highlights fundamental problems with password policy, systems management and auditing.”
“All of these failures suggest an organization rife with systemic security issues, an ineffective security management program, and a lack of technical controls or compliance with industry standards," Rogers continued. "By focusing on the fact that an intern leaked the password through their private GitHub, they are also clearly still missing the point. Yes, that event was troubling, but the journey it took to get there was littered with failures and missed opportunities that would have prevented it from ever happening in the first place.”
Asked about “solarwinds123” during last Friday’s Congressional hearing, former CEO Kevin Thompson called the password “a mistake that an intern made. They violated our password policies and they posted that password… on their own private GitHub account. As soon as it was identified and brought to the attention of my security team, they took that down."
Current SolarWinds CEO Sudhakar Ramakrishna, who replaced the recently retired Thompson on Dec. 7, 2020, similarly testified that an intern set the company password on one of his or her GitHub servers back in 2017. In all that time, SolarWinds’ credentials never changed.
“So an intern who worked for only 3 months (2017) had an access to the FTP server and credential was not rotated after he left. So solarwinds123 is the password for more than 2.5 years,” tweeted independent researcher Vinoth Kumar, adding a laughing-so-hard-I’m-crying emoji. It was Kumar who discovered the exposed password, which was accessible online since at least June 2018, up until SolarWinds corrected the issue in November 2019.
The earliest suspicious activity tied to the SolarWinds supply chain Sunburst malware attack took place in September 2019, prior to the server's password getting taken down from GitHub. However, no connection to the SolarWinds attack and the leaked password has been established so far. Moreover, a statement that SolarWinds supplied to SC Media said that the password was actually for a third-party application that was not connected with SolarWinds’ IT systems – though this was reportedly not mentioned during the public testimony.
"We have determined that the credentials using that password were for a third-party vendor application and not for access to the SolarWinds IT systems," the statement reads. "Furthermore, the third-party application did not connect with the SolarWinds IT systems. As such, we have determined that the credentials using this password had nothing to do with the Sunburst attack or other breach of our IT systems."
The password gaffe exposed SolarWinds to ridicule from Rep. Katie Porter, D-Calif., who told Ramakrishna: “I’ve got a stronger password than Solarwinds123 to stop my kids from watching too much YouTube on their iPad.”
Infosec experts similarly chided the company for a lack of strong credentials.
“The latest developments in relation to the SolarWinds intern’s poor password choice highlight’s how bad password hygiene is getting and how important it is for organizations to prioritize password management,” said Joseph Carson, chief security scientist and advisory CISO at Thycotic.
“Password hygiene should be part of employee training and cyber awareness training,” Carson continued. “Organizations must help employees move passwords into the background so they do not have to choose or remember passwords.” That way, they don’t make classic mistakes like implementing weak or recycled passwords, or even slightly altered variations of common or reused passwords.
“Many password managers are free,” said Carson. “Use unique long passwords such as passphrases, and use a password manager to keep all your passwords unique but easy...”
As noted by Kumar in his tweet, SolarWinds also made a grievous error by not rotating its passwords. “By admitting the password was actually implemented in 2017 and not changed until 2020, the former CEO of Solarwinds made it abundantly clear that these issues were likely long standing and systemic,” said Rogers.
There’s also the question of how much network access low-level, temporary interns should have been granted in the first place. Rogers called it a “complete failure to either implement or enforce role-based access control (RBAC)," asking "What other production systems did this intern, or others at that level, have access to?"
"In my experience, organizations that allow junior employees privileged access to production systems like this are typically a ‘Wild West’ when it comes to controlling access for all systems, not just one.”
“Any company with an effective role-based security model, technology that enforces RBAC, and rigorous auditing of user access logs won’t need to consider interns’ activities because that particular problem will have already been addressed,” Rogers continued.
Instead of or in addition to role-based access, companies could also take a risk-based approach, placing the most access controls on their crown-jewel assets – the ones that would generate the most severe consequences if they were breached and accessed, said Brandon Hoffman, CISO at Netenrich.
“Additionally, understanding identity and controlling access from a federated standpoint would have also prevented this issue,” Hoffman continued. “Both of these tasks are basic security processes that should be put in place before other more complex controls are implemented. It is likely that SolarWinds has these processes, but perhaps they were not updated on the required frequency or something slipped through the cracks.”
The need for such controls highlights the importance of concepts such as identity and access management (IAM), privileged access management (PAM) and zero-trust policies.
“Identify and access management is the dirty work performed down in the trenches of our cybersecurity programs,” said Rick Holland, CISO and vice president of strategy at Digital Shadows. “The concepts of least privilege and multi-factor authentication aren’t exciting, but essential. Enterprise-wide IAM is a challenge with disparate systems, but should be a top priority.”
"The authentication lesson, plain for all to see and act on, is to implement privileged access management to compensate for lapses in password judgment, require multi-factor authentication for all logins, using cryptography to create unforgeable integrity and authenticity checks, and monitoring the heck out of your most important functions and assets to look evidence that you’ve been compromised," said Phil Quade, CISO at Fortinet. "And network access control (NAC) can allow you to bring-in temporary and permanent employees – interns or otherwise – with confidence by automating the enforcement of access, with the agility that’s needed in the age of users and devices on the move.”
Of course, even with better passwords and access management, incidents will still occur, which is why companies must also focus on resilience and mitigation to avoid becoming the next SolarWinds. “Bad passwords will be selected, and inevitably may leak," said Tim Wade, technical director of the CTO Team, at Vectra. "Success is detecting, responding and recovering from such an event before material damage is done, not going on a fool’s errand to stop interns from acting like interns.”
“So while, yes, the policy and controls necessary to protect against poor password selection and leakage are valuable, what’s more telling is that there seems to be the expectation that security will be capable of eliminating human error. It won’t, and yet we need to be secure despite that.”
Ultimately, when a cyberattack does happen, the victim company and its leaders must accept responsibility and accountability, the experts said. That means not making an intern a scapegoat.
“This is not an intern problem, but rather a management problem,” said Rogers. “Organizations should… consider the long-term impact of blaming junior staff members for failings of this magnitude. A key component of any successful security program is trust. As security leaders, we trust that our employees will come forward when incidents occur, and our employees trust that we won’t shoot the messenger or punish them for our collective failings. Without that trust any security program is a castle built on sand.”
“The... buck stops here analogy is appropriate,” said Holland.” Sarbanes-Oxley established CEO and CFO accountability for financial records, but responsibility needs to expand beyond that. The CEO is responsible for any environment that enables an employee, an intern or contractor to make a mistake. We need more CEO accountability and less victim-blaming.”
"One hundred percent it was a bad showing,” said Hoffman. “There is a significant disconnect between business management and security. Having strong security awareness would be when upper management understands that a breach cannot be pinned down to a single individual’s actions – mostly. If appropriate controls were in place then the action a single person, especially and intern, would not have created such a large issue.”
Senior Reporter Joe Uchill contributed to this report.
