The MITRE Engenuity ATT&CK evaluations, or Evals for short, are a step-by-step analysis of how leading endpoint security products fare against well-known adversaries. Unlike traditional evaluations that can tell you only whether a threat was stopped, the Evals map out every step of a kill chain, giving security vendors, security customers and third-party defenders alike an unparalleled view of how to defend against attacks.
The Engenuity evaluations examine how leading endpoint security products protect systems from well-known threat actors. The results are based on the MITRE ATT&CK framework and organizations can use them to see how security products will mesh with their own systems — and how far threat actors can penetrate those systems. Security vendors can also use the results to improve their own products.
The MITRE ATT&CK framework
Underpinning the Engenuity evaluations is the MITRE ATT&CK framework, a well-understood and widely used method for analyzing and planning against cyber threats and attacks.
Released to the public in 2015, the ATT&CK (for Adversarial Tactics, Techniques, and Common Knowledge) framework consists of matrices, or two-dimensional checklists, that contain constantly updated data about attacker tactics, techniques and procedures (TTPs) and are freely available on the MITRE ATT&CK website.
The ATT&CK framework is "largely a knowledge base of adversarial techniques," explains a MITRE blog post. "The focus isn't on the tools and malware that adversaries use but on how they interact with systems during an operation. ATT&CK organizes these techniques into a set of tactics to help explain to provide context for the technique."
As of this writing in June 2022, ATT&CK is up to version 11 and there are ATT&CK matrices for enterprise, mobile and ICS environments. The enterprise matrix can be broken down into specific matrices for Windows, macOS, Linux, cloud, networks and containers, and the cloud one can be further subdivided into different cloud environments. Each "parent" matrix contains all the tactics and techniques from its "child" matrices.
How the Engenuity evaluations use the ATT&CK framework
The MITRE Engenuity ATT&CK evaluations, which began in 2018, transparently assess the efficacy of commercial endpoint protection products against simulated attacks by well-known threat actors. Instead of merely telling you whether a particular product blocked or neutralized a specific attack, the Engenuity evaluations use the ATT&CK framework to reveal how a security product fares at each step along the kill chain.
Those attacks are launched against Microsoft Azure cloud instances and defended by real-life endpoint security products. Each kill chain is divided into multiple segments with several steps each.
Regardless of whether an attack is blocked during one segment, the next segment assumes the attack upon the previous segment was successful and the attack resumes. In this way, security products that block attacks early in the kill chain must demonstrate how they defend against subsequent tactics and techniques.
In the 2022 Evals, 30 different endpoint security products were tested, including entries from Cisco, CrowdStrike, McAfee, Microsoft, Palo Alto Networks and Symantec. The products had to defend Windows (and, optionally, Linux) systems against simulated attacks by the Wizard Spider cybercriminal group (known for deploying the Conti, Ryuk and TrickBot malware) and the Sandworm state-sponsored group (best known for unleashing the NotPetya wiper worm in 2017).
The results of the evaluations are posted online, but MITRE does not assign scores or rank products tested. There are no "winners" or "losers." The Engenuity evaluations are meant to show how an endpoint security solution works rather than how well it works.
"Each vendor evaluation is independently assessed on their unique approach to threat detection," states the MITRE Engenuity ATT&CK website. "Evaluation rounds are not a competitive analysis; they do not showcase scores, rankings, or ratings and are transparent and openly published."
How the Engenuity results help vendors, customers and others
The ATT&CK framework is widely used — a recent study found that 81% of enterprises in the U.S., U.K. and Australia use it. The results from Evals offer a very detailed look into how a particular security product fares against specific threat actors.
Security teams that are familiar with their own organizations' ATT&CK profiles can see whether an endpoint product fills the gaps in their own security postures. "By viewing the MITRE ATT&CK framework as a 'board game' or checklist, security teams can thoroughly understand where their vulnerabilities lie and take the appropriate action to prevent attacks," said Dr. Joel Fulton, co-founder and CEO of Lucidum, an asset discovery company.
Even organizations that are not considering purchasing a security product can use the Evals results to spot potential weaknesses in their defenses.
"Understanding the ways a potential attacker will take advantage of an organization is critical in being able to repel those very same attackers, and if not repel, identify more quickly when they have been successful," said Dave Cundiff, CISO at Cyvatar.
Meanwhile, security vendors can use the Engenuity Evals results to see exactly where their products fell short and quickly work to improve performance.
For example, Cisco found that its Endpoint Security Advantage did not stop Sandworm from installing command-and-control malware and gaining persistence on a Linux server. Shyue Hong Chuang, product manager for Cisco Secure Endpoint, said his team knows what to do as a result.
"We're going to increase our ability to mitigate living-off-the-land abuse by introducing more advanced behavioral protection on the Linux platform," Chuang said. "We believe when we introduce behavioral protection into the Linux platform, we'd be able to see these events firing and kick in to kill that process."
Perhaps most importantly, the ubiquity of the ATT&CK platform will help CISOs and other security staffers explain to non-technical colleagues, including executives, which potential security products might be the best fit for their organizations, and also where their organizations' cybersecurity strengths and weaknesses lie.
"Most CISOs will ask for investments and increases in budget to respond to either current events or longstanding security concerns, but they don't have sufficient data points to support the ask," said Fulton. "By using the MITRE ATT&CK framework as a guide for these conversations, CISOs will be able to effectively explain the severity of threats and the actions to mitigate them while allowing CIOs to be active participants."
