In the aftermath of the compromise and attempted sabotage of the Bruce T. Haddock Water Treatment Plant in Oldsmar, Florida earlier this year, threat analysts at ICS security company Dragos conducted an investigation into the incident – and for a brief moment, it appeared as if they had discovered a bombshell.
Dragos discovered a watering-hole attack that had compromised a website – operated by a Florida-based water utility contractor – that had been infecting visitors with malicious code. Moreover, a user at the Oldsmar plant had actually visited the site on the very day of the attack. The discovery set off alarm bells – yet Dragos, as it explained in this company blog post – ultimately determined that its discovery was unrelated to the Oldsmar incident, in which an actor exploited TeamViewer to hijack plant controls and then tried to increase the amount of lye in the water to dangerous levels.
According to Dragos, the purpose of the malware distributed in the watering-hole attack appears to be to collect data on victims (e.g. operating system, browser, presence of camera and microphone and more) and conduct fingerprinting on them. Dragos believes the motivation behind these actions is simply to train the attackers’ botnet, called Tofsee, to better imitate legitimate browser activity so its future behavior will look more realistic to cyber solutions.
In the words of Sergio Caltagirone, vice president of threat intel at Dragos, the scheme turned out to be a “nothingburger.” Nevertheless, there is a valuable lesson to be gleaned in the way Dragos handled the event. The company found the discovery concerning enough to disclose to relevant parties, but was careful not to alert the general public – not when it could not confidently attribute the Oldsmar incident to the watering-hole attack.
Striking the balance between offering timely information and complete information was key in this case, said Caltagirone, who in an interview with SC Media offered a recap of the company’s investigation and disclosure process.
Following the Oldsmar incident, the Dragos team searched for clues or activity related to the attack. Take me through the investigatory process and what led to the discovery of the watering-hole attack.
We were examining global internet traffic telemetry. We have sources of data that show us flows of data around the internet. And so we look for anomalous and interesting activity in there that comes from potential victims. If we find a victim of something happening… we will then pivot into what's going on around that incident to see what we can find. So we were looking at things like anomalous RDP traffic… we were looking for interesting traffic around TeamViewer… and eventually we went through our first run of hypotheses were like, “Well what else is here?”
Then what we do is something called… subtractive analysis, which is where you actually remove everything that you think is interesting and you look at what's left. We started looking at what was left in the data and we started finding these flows from Oldsmar in the same time period [of the attack] to a server sitting in Switzerland. And we were like, “What's going on here?. That's just weird. Why is Oldsmar talking to this server in Switzerland?”
We looked into it and we found out that they were going to this website. We then talked to… not only just the owner of the website but the hosting service as well. They gave us a whole bunch of data, and that's when we found out there's this heavily obfuscated JavaScript code sitting on this website that hasn't been touched in three years. And Oldsmar happened to touch this the morning of the attack.
Obviously, alarm bells went off. Because… we know adversaries in our ICS space are leveraging watering holes.
Once you discovered this, how did you tackle disclosure?
Immediately, we called a couple of water utility associations, and included some states and some national organizations and so forth.
We basically wrote up a quick summary: “Here’s what we found, here's the website. If you're touching this, we don't yet know what this is, but… you should probably look into it.” And we always do that. We try to provide people early warning before we may not know everything that's happening. And that way, of course, it can prevent anything worse.
Then it took us about two weeks to reverse-engineer all the JavaScript code. It was really heavily obfuscated. And we found that it was using these odd fingerprinting techniques to capture the encrypted handshake protocols that were being used by the visitors.
It was capturing things like browser details, what extensions were installed, and things like that. And it was all being sent back to a server being hosted in a Salesforce app – the Heroku app store, which is kind of like a virtual server. They were storing it inside of this PostgreSQL database. We contacted Salesforce and we worked with them on the back end of that stuff, which is where we got the telemetry of everyone who was talking to the script.
And then we [followed the threat back to a] dark web marketplace. We ended up uncovering that [the attackers compromised] a WordPress site that hadn't been updated in a while, and they hosted their script on it and they were collecting browser fingerprints to basically masquerade their botnet traffic, so that they could get past Palo Alto and Cisco and all the firewalls that now can more intelligently filter all that traffic out.
We reported it publicly, even though it was kind of a nothingburger to some extent, as more of an education piece that watering holes are out here – they hit these niche sites.
It may have been a nothingburger, but wouldn’t you agree there is still a valuable lesson for cyber organizations about the importance of due diligence, and practicing responsible reporting and attribution – especially considering that the watering-hole attack was a red herring?
In a day and a half, we thought this massively interesting thing happened that correlated to [the Oldsmar attack]… Sometimes you think you landed on a diamond mine… And then we end up spending about a month of work ripping all of this apart, finding out who's behind it, why they were doing it, and we ended up like, “Well this was just… a very interesting coincidence.”
Intelligence is critical… I think there's two things you've got to balance. You have to balance completeness and timeliness, and usually those things don't come together. Usually, if you want to be fast, you can't be complete. If you want to be complete, you can't be fast. It's one or the other in intel.
And so in our world, where we feel that there is an immediate need for defenders to take action, we will sacrifice completeness for timeliness. But what we don't tend to do is public disclosures. And the reason for that is, it ends up muddying a lot of the water… I love the work that reporters do but it causes the reporters to [sense] blood in the water and everyone jumps on [it]. And one of the things my customers hate is, instead of protecting their network, having to now respond to media requests or work with executives who are being questioned by their board or media.
One of the things we try to balance is the frothiness of cybersecurity. When you create these stories, there's this huge froth that exists, and it doesn't allow defenders the space and time they need to do the research in defense of the environment. And so that's one of the reasons why, even though this could have been a gigantic thing, we didn't go public. We went to the water utility associations and we were like, “hey, here’s what's going on, here's what we know.” We avoided the public notification. Even though we may have gotten to more people, it probably would have caused more damage. It would have made our job harder, it would have made the defender's job harder.
[Once we learned more,] we said, “okay, now that we know something one way or another, let's go public and share the story of how all of this works." So it's a balancing act. The question we always have in our mind is: How will the information and intelligence best get to the people who can do something? And if it's through the media – if I need to tell the media this because more electric utilities around the world will listen than if I talk to 20 governments today – then sometimes we make the decision that the media is the best outlet to inform the broader community.
After you discovered that the malware’s chief functionality was fingerprinting, the next step I imagine was to determine what the attacker’s likely motivation was. As you were going through the possible scenarios, what made you lean toward the theory that the attackers were simply trying to improve their botnet as opposed to some grander scheme whereby the fingerprinting was one of several tools used in a potential sabotage incident?
We can never rule out all hypotheses in any sort of work. But we take an alternative competing hypothesis approach… You build evidence either for or against each of these hypotheses, and then you get to a list of… the most likely, and then you end up [ranking] your hypotheses.
Eventually, as we were digging deeper and deeper and deeper, we ended up finding the botnets responsible for what was going on here, and how they were using it. And we worked with other security companies and utilities in the space that saw more than we did on what the botnets were doing, and it became very clear.
Of course, the challenge here is that botnets, by their very nature, just provide general level access. They could be leveraged in any number of ways to compromise a victim… They could start off as [a minor] adversary and then all of a sudden they find out that they're inside of water utility and they're like, “Hey I know somebody in my government who would buy this type of access.” And then all of a sudden they become much more interesting to people than they did as a botnet. So we are always on the lookout for that.
But the problem is that those are still very rare events, so we have to balance the likelihood of it happening with the actual impact.
Tell me more about the dark web marketplace that you discovered was also subverted by the same Tofnee botnet actor. What did you learn about the attackers from that?
The dark web website we understand was a front end for the controllers. So that's where they were basically selling access to this capability. The work we did was in conjunction with several other security companies who… were able to see what the botnet was doing, what data they were going after, what they were using the botnet for, and who they were selling it to.
We recognized that the [attacker] profile didn't fit anybody who was going to target water. This was general commodity crimeware kind of stuff going on. It had nothing to do with ICS. They obviously didn’t want attention. They weren't trying to blow something up, so to speak, because they would ruin their own operations. It was a very small operation, it was tiny. Overall, when we talk about actor motivations and intent, nothing else lined up with what you see in terms of attacking water as a critical utility.
Most malicious operators out there have no interest in notoriety, and in fact it harms their operation… because it just puts too much heat on them. This is one of the reasons we hypothesize that crimeware doesn't generally target ICS.
