A malicious hacker’s attempted poisoning of the Oldsmar, Florida water supply serves as a stark reminder of the potentially devastating consequences that can result from operating vulnerable and unsecured industrial controls in a critical infrastructure environment.
Oldsmar and Pinellas County, Fla. officials today revealed that an unknown individual last Friday morning hijacked a remote access system used by employees at the city’s water treatment plant. The hacker attempted to increase the amount of sodium hydroxide in the water from 100 parts per million to 11,100 parts per million. Sodium hydroxide, which is found in drain cleaners and is commonly known as lye, is used to reduce the acidity of water and make it more potable – but too much of it makes the water caustic and potentially deadly.
Water and wastewater treatment is among the most at-risk areas of critical infrastructure that exists today, said Grant Geyer, chief product officer at Claroty. He pointed to the company's Biannual ICS Risk & Vulnerability Report, which found that industrial control system vulnerabilities disclosed during the second half of 2020 increased by 54% from the second half of 2019 and 63% from the second half of 2018 in water and wastewater.
Geyer attributes the growing number of bugs to “the long depreciation period of equipment in critical infrastructure environments” as well as “technology obsolescence.” Moreover, “many water utilities are small entities and are under-resourced, making the challenge of developing a robust security program that much more challenging.”
Austin Berglas, global head of professional services at BlueVoyant, agreed that water facilities’ ICS and SCADA systems are “outdated, unpatched, and available for review on the internet, leaving them incredibly vulnerable to compromise.”
“In addition, many ICS solutions were designed for non-internet facing environments, and therefore did not incorporate certain basic security controls. This offers additional vulnerabilities as more and more operational technology environments are allowing access to their ICS systems from the internet,” continued Berglas, who, as former FBI assistant special agent in charge of cyber investigated the 2013 compromise of the Bowman Avenue Dam in Rye Brook by Iranian hackers.
And Marty Edwards, vice president of OT at Tenable, said more vulnerabilities are accumulating as OT networks become less and less isolated from IT systems. These days, security professionals in critical infrastructure environments must contend with “a highly dynamic and complex environment of smart OT technology, modern IT and everything in between,” said Edwards, a former director of the U.S. Department of Homeland Security’s Industrial Control Systems Cyber Emergency Response Team (ICS-CERT). “Attackers have capitalized on these converged networks to move laterally from one system to another, making the compromise of just one device even more dangerous.”
In a press conference, Pinellas County Sheriff Bob Gualtieri said that the remote access system is used by its supervisor and other workers to “troubleshoot system problems from other locations.” But on Friday, a plant operator noticed two successful attempts at remotely accessing the system that controls chemical dispersal. On the second attempt, the employee noticed “a mouse being moved about to open various software functions to control the water being treated in the system.” When the intruder attempted to change the chemical composition, the employee acted swiftly and “immediately reduced the level back to the appropriate amount.”
Officials at the plant said they have since disabled the remote access system – and plan to make secure upgrades to additional systems. But should remote access have been enabled at all in a critical infrastructure environment? According to experts, there’s little choice but to do so – so it’s imperative that such tech be instituted responsibly.
“The nature of our increasingly digitized world, especially with the shift to remote work caused by the pandemic, makes remote access a requirement – even in critical infrastructure,” said Geyer. “This isn’t a ‘should we or shouldn’t we?’ discussion... The key is how remote access can be implemented securely – with strong authentication mechanisms, access controls, auditing, and session recording.”
“There is a justifiable reason for providing remote access,” agreed Mike Hamilton, president and chief information security officer at CI Security and former CISO of Seattle. “But enabling that access in the absence of security requirements invites these types of episodes. If remote access is a requirement, the water and other critical sectors should enable it only when needed, audit its use frequently, and ensure that multi-factor authentication is used."
Dragos, which provides solutions to help secure industrial networks, issued a similar statement, noting that “remote access to industrial control systems is common and increasingly so due to the need for people to work remotely. This incident underscores how important it is for asset owners and operators to assess and secure their remote connections, especially internet connected remote access, and to ensure their incident response plans are current.”
No one should presume this is a fluke. In fact, while this incident quickly garnered significant attention, FireEye’s Mandiant division has reported that a series of low-complexity, non-impact incidents against critical infrastructure in targets have recently taken place under the radar.
“Since last year, Mandiant Threat Intelligence has observed an increase in cyber incidents perpetrated by low sophisticated actors seeking to access and learn about remotely accessible industrial systems,” said Daniel Kapellmann Zafra, manager of analysis at Mandiant Threat Intelligence. “Many of the victims appear to have been selected arbitrarily, such as small critical infrastructure asset owners and operators who serve a limited population set. Through remote interaction with these systems, actors have engaged in limited-impact operations that often included manipulation of variables from physical processes.”
Mandiant believes one reason behind these recent incidents is that the barrier of entry for unsophisticated actors to attack industrial controls is lower due to the “increased availability of tools and resources that allow malicious actors to learn about interact with these systems.”
The incidents tracked by Mandiant have not resulted in damage due to the presence of additional safety mechanisms and workers who monitor OT systems for anomalies, Mandiant noted. Indeed, local city and county officials said the water plant had its own safeguards and redundancies in place that would have prevented a disaster even if the hacked hadn’t been immediately noticed by a worker. This includes alarms that go off when a change in pH is detected. Additionally, it takes 24-36 before the affected would have reached the water supply, allowing plenty of time for redundancy mechanisms to detect an attack.
But Hamilton noted that plant operators cannot afford to be complacent, because a more sophisticated attackers could have potentially tampered with some of those redundancies as well.
“Other compromises of industrial control or 'SCADA' systems have manipulated the status screens of the human machine interface, showing that everything was operating normally,” said Hamilton. “The fact that this wasn’t done here is suggestive of a crime of opportunity,” by less sophisticated actors.
“Other systems are equally vulnerable,” agreed Ron Brash, director of cybersecurity insights at Verve Industrial, such as systems that analyze water for heavy metals, for example. “And they’re very rarely protected very well.”
“If I were a [municipal utility] CISO, I’d be doubling down on cybersecurity basics,” said Brash. “But they may not have the budget to do what they need to do.” The good news is, “These water-type environments don’t change very often. They’re fairly static, steady state environments, so at least you’re in a very good defensible position.”
SC Media Senior Reporter Joe Uchill contributed to this report.
