The MITRE Engenuity ATT&CK evaluations of enterprise endpoint security products provide so much detail that vendors use the results, based on the well-known MITRE ATT&CK framework, to pinpoint the weaknesses in their own offerings. Cisco's experience with the 2022 Engenuity evaluations provides an example.
How the MITRE Engenuity ATT&CK evaluations work
Traditional antivirus evaluations merely tell you whether a particular exploit is blocked or neutralized by the antivirus product being tested.
MITRE's yearly Engenuity ATT&CK evaluations of endpoint security products, or Evals for short, go into much more detail. They use the MITRE ATT&CK framework to examine every link of a well-known attack kill chain step-by-step, from initial access to final goal.
The Evals results can tell you, for example, whether a particular endpoint security product succeeded in blocking privilege escalation or failed to block password harvesting. Highly successful threat actors such as those emulated by the Engenuity tests use attacks involving dozens of steps, and the Evals results track the outcome of each step.
Because the results and methodologies of the evaluations are freely posted on the MITRE website, organizations that are considering new endpoint security products and are also familiar with the MITRE ATT&CK framework can pore over the results to see how well specific offerings did, and which might be the best fit for their individual security postures.
Security vendors also go over the results of the Engenuity Evals to see where their products fall short — and how they can be improved for the next round. Cisco's own experience shows how the MITRE Engenuity ATT&CK Evals can raise the bar for all makers of endpoint security products.
How Cisco fared in the most recent MITRE Engenuity evaluations
Cisco's Endpoint Security Advantage was one of 30 different products tested in the 2022 round of Engenuity Evals, alongside offerings from Bitdefender, CrowdStrike, FireEye, McAfee, Microsoft, Palo Alto Networks, Rapid 7 and Symantec.
Within the test environment of a Microsoft Azure cloud instance, each endpoint security offering in the most recent round of evaluations was pitted against the most common attack scenarios of two very well-known adversaries. First up was Wizard Spider, a Russian-speaking cybercrime group known to deploy infamous malware such as Conti, Emotet, Ryuk and Trickbot against organizations.
The second simulated adversary was Sandworm, aka Black Energy, a Russian state-sponsored group that first came to light attacking the Ukrainian energy sector in 2014. Sandworm is best known for the NotPetya wiper malware that spread around the world in June 2017 — and indeed, the end goal of the Sandworm simulation in the Engenuity Evals was the deployment of NotPetya.
Cisco's 2022 MITRE Engenuity results were good. Cisco Secure Endpoint Advantage detected Wizard Spider activity in 10 out of 10 steps, and Sandworm activity in 9 out 9 steps, for an overall 100% detection rate.
A third-party overview of the 2022 MITRE Engenuity ATT&CK results found that some other security products had detection rates as low as 57%. Prevention rates varied from nearly 90% to less than 4%, and eight of the 30 vendors opted out of the prevention scores.
Yet Cisco's results were not perfect. Its prevention rate was only 78% due to two noteworthy compromises:
- First, the Wizard Spider simulation managed to dump the Active Directory database in the fourth segment of its attack against Cisco Secure Endpoint Advantage. However, because each independent segment in the simulated attacks assumes successful compromise of previous segments, it's likely that a real-world attack would have been stopped earlier. Cisco blocked the Emotet initial access that made up the first part of the first Wizard Spider segment.
- A second, more serious compromise took place in the first segment of the simulated Sandworm attack. Cisco failed to block an attacker using stolen credentials from installing command-and-control malware and gaining persistence on a Linux server. That's game over, right away, even though Cisco did very well defending both Linux and Windows systems against the rest of the Sandworm attack.
“There could have been places where Wizard Spider could have been blocked before it even got in," added Adam Tomeo, senior product marketing manager for Cisco Secure Endpoint. "Because this is an email compromise, Cisco Secure Email might have picked it up as well before it even had a chance to come in."
Lessons learned and planned improvements
For Shyue Hong Chuang, product manager at Cisco Secure Endpoint, this was a teachable moment. He vowed to use the latest Engenuity results to bring Cisco's defenses for Linux up to par with its Windows defenses.
"We're going to increase our ability to mitigate living-off-the-land abuse by introducing more advanced behavioral protection on the Linux platform," Chuang said. "It's something we have seen extensively in the Windows world [and] we're now going to come around and double down to bring that technology into the Linux platform as well."
Indeed, Chuang said Cisco has already implemented the lessons learned from the 2021 MITRE Engenuity ATT&CK Evals, in which Cisco's and 28 other vendors' endpoint security products faced off against simulated Carbanak and Fin7 cybercriminal attacks.
He cited a subsequent threefold improvement in Cisco Secure Endpoint's ability to deliver analytic detections across kill chains, which Chuang chalked up to improved MITRE ATT&CK Tactic, Technique and Sub-technique mappings, enhancing the product’s behavioral-protection capabilities and exposing behavioral telemetry to customers.
With this success, Chuang is confident that Cisco will be able to quickly improve its Linux defenses as well.
"We already have all these detections [in Linux]," he said. "Some of these come across to us as very high confidence, but we need a mechanism to kick in to kill that process. And we're going to build that mechanism now."
