Five takeaways from the Oldsmar water facility attack

The attempted attack recently on the Oldsmar, Fla., water facility highlights the threats and vulnerabilities facing our nation’s critical infrastructure, including the more than 50,000 water production utilities across the United States.  If an unsophisticated attacker with a few mouse clicks can start the process of poisoning our water supply, then what could a medium or highly-skilled attacker do?

This incident reflects the state of all too many industrial control system (ICS)  installations, especially those with smaller budgets or size, where security gets overlooked or under deployed. Attackers aren’t going away, so we need to defend against remote access and supply-chain risks for the foreseeable future. How can we minimize the impact of these risks?

Start by focusing on process. For water facilities we need to ask: How does the water get produced? What specific operation technology (OT) processes are involved? What are the risks to those processes? If we start with the process, we learn that some devices are more worthy of our attention than others. Does the engineer need to secure his iPhone or the digital control panel in the operations center?  By focusing on process and industrial controls, we can detect threats faster and mitigate security risks.

In the spirit of learning from the Oldsmar incident, here are five steps to help protect critical infrastructure from attack:

For many critical infrastructure facilities, COVID-19 forced an abrupt shift to employees working from home. This meant that security teams had to make production control networks accessible remotely to keep systems up and running. Unfortunately, as seen at Oldsmar, remote access apps like TeamViewer are often the easiest path for attackers with stolen credentials to infiltrate a network. Managers need to secure their organization’s remote access by adding endpoint protection, using good password hygiene, installing network firewalls, and most important, continuously monitoring their remote activity. It appears that almost none of this was done at Oldsmar. With the right monitoring tools, security teams can quickly identify anomalous activity – such as abnormally high number of remote connections, the use of unusual protocols in those connections, and atypical behavior of the remote user -- before operations are disrupted.  

If the security team doesn’t have visibility into all of the devices on their network, they can’t protect their assets or segment the network for better resiliency. It’s not uncommon for organizations to think that they have 5,000 devices, when it’s actually twice that. By taking an inventory of all network assets, security teams can achieve real-time visibility into their devices, connections, communications, and protocols. This visibility will let security pros monitor, identify and troubleshoot networking and communication issues that threaten reliability. It’s important to have an accurate, centralized asset inventory for effective cybersecurity and operational monitoring.

Industrial networks contain thousands of OT and IoT devices from a variety of vendors. Unfortunately, most of those devices aren’t designed for the level of security required in a critical infrastructure environment. Many ICS devices are insecure by design – lacking authentication, encryption, and other security standards that typically apply to IT applications and systems.  All too often, attackers exploit known, but unpatched vulnerabilities for malicious activity. It’s also risky to use software that’s reached end-of-life, such as the Windows 7 systems found on the Oldsmar network, as it’s unsupported and won’t receive security updates for newly discovered vulnerabilities. Tools that automatically identify system vulnerabilities, utilizing the National Vulnerability Database (NVD) for standardized naming, description and scoring, can help rapidly determine which devices are at risk, prioritize, and recommend firmware updates or other specific remediation.

Protect processes by monitoring the actual industrial process tags and values to look for anomalies within. Automated network anomaly detection solutions leverage artificial intelligence to run anomaly detection against the actual parameters that are used to control the industrial process. 

Take a pump, for example. If it’s rated to spin at 100 rotations per minute, it’s not safe to run it higher. The engineers that programmed the Human-Machine-Interface (HMI) shouldn’t let dangerous conditions be present, and in most cases would prevent operators from entering invalid inputs or introducing unsafe conditions like sending a parameter value to the pump to spin at 150 rpm. When security teams monitor the process, this and many other risks, such as negligent or compromised insiders, are mitigated. So, if someone purposely or accidentally sends a pump at 150 rpm, the security solution will detect it.

In the case of Oldsmar, the attack happened within the stream of data used to monitor and control the process. The attacker used legitimate (albeit possibly stolen) credentials and HMI to send a legitimate packet with a legitimate payload which increased the quantity of sodium hydroxide in the water. But if anomaly detection was applied to the OT parameters in the Oldsmar facility, the attack would have been detected and blocked.

OT knows how to meet production targets and keep the plant running safely, while IT can address networking and cybersecurity issues that are unfamiliar to ICS staff. When IT and OT work together, we see stronger operational resiliency. Unfortunately, oversight of OT security is often still quite fragmented. IT and OT must collaborate to reduce the blind spots and security risks surrounding highly connected industrial control systems.

One final thought:  Oldsmar has something important in common with the many other OT-specific attacks we’ve observed. These attacks all impacted the process, and used the approved technology infrastructure to target that process. When the infrastructure gets targeted, security teams can detect attacks more easily, but when the infrastructure gets used to attack the process, the situation becomes more complicated, and more sophisticated tools are required to mitigate the threats.

More than ever, our nation’s critical infrastructure and industrial facilities are prime targets for bad actors and nation-state attackers. Build your cybersecurity program around these five tenets, and the organization will find itself much better equipped to defend against process-based attacks and improve operational resiliency.

Andrea Carcano, co-founder, Nozomi Networks