Attack simulation is the technology that enables use cases in this market. In short, it can answer some of the most elusive and sought-after questions in enterprise security, like:
- How secure are we?
- If we got hit with a targeted attack today, would our staff see it?
- Are we monitoring and alerting on the right things?
- Could we respond to a threat quickly enough to make a difference?
- Could we contain and clean up the threat effectively?
- (and if vendors are comparing customer data:) How do we compare to our peers?
It is a sort of Question Answerer for some key security questions. It separates reality from fantasy. Replaces assumption with fact. A common trope in books and movies is a device or an animal that helps the protagonist see past glamours. In this age of vendors offering simple solutions to complex problems, defenders need the ability to see past the glamour of marketing.
Or perhaps The Emperor’s New Clothes is the better analogy?
Breach and Attack Simulation (BAS)
This acronym was coined a few months ago by Gartner, but commercial examples of attack simulation products have been around for five or more years. Open source options have become common as well. As a general term, it refers to the ability to simulate adversarial activities with some degree of automation. The result, as you could imagine, is a wide range of use cases and products.
Side note: I’m not associated with Gartner at all, but they recently did a lot of research on this market and authored some excellent blog posts on the topic. I’ll be referencing more than one of those posts.
On the commercial side, the vendors I’m most familiar with include AttackIQ, Cymulate, SafeBreach, Threatcare, and Verodin. The workflow for most of these products is similar—run tests, review results, fix the gaps revealed by the tests and repeat the process.
Now, a BAS tool could be used as part of this workflow could until a positive result is achieved, then discontinued. However, most of these products are designed to continue testing indefinitely. The reasoning is that, even if security controls are working today, any change or update to the environment could potentially change this.
When Security Products Fail: Functional Failures
I’m reminded of an incident early in my career, when some battery back-up maintenance required powering down everything in the data center, including the SIEM. We weren’t asked to participate, because most of our servers would boot on their own after power was restored to the rack. Not so with our SIEM appliances. They required a physical button press to power back on. We realized this three weeks later when we had an incident and tried to log into the SIEM to check the logs.
Lesson learned: we needed some sort of uptime monitor, at the very least. This is an example of functional testing and it’s the easiest problem that BAS can address. It’s more common than most might think. There are plenty of similar stories of SIEMs failing to collect logs, security appliances plugged into inactive network taps and inline protection that turns out to be… not so inline. Implementation should be a simple process but with security products, is often not validated.
The problem is that most technology fails in obvious ways. Since most security products are designed to prevent or detect threats, they fail silently. Hence the need for validation when products are implemented and continuously throughout their life cycle.
When Security Products Fail: Efficacy
So what does a breach and attack simulation product need to be effective? Not much for functional controls testing—the equivalents of EICAR across the board can do that. Creating simulations that resemble what attackers are currently using is a different challenge altogether. There are three factors at play here: breadth of coverage, depth of coverage and the sophistication of the attack (e.g. its ability to evade discovery or defeat controls).
Breadth of Coverage
Breadth refers to the need to simulate attacks from different perspectives: both network and on hosts. On the corporate network and off. In the cloud, even.
Depth of Coverage
Depth refers to how ‘complete’ the testing is. While an intrusion detection sensor could be functionally tested with a single simulation, a larger battery of tests are necessary to test its full functionality and its efficacy against current attack techniques.
Sophistication
Sophistication refers to cycle of attackers improving the ability to bypass defenses, evade detection and remain unnoticed. Simulations also must imitate and innovate whenever possible for BAS products to remain relevant to security teams, given the current state of attacker tactics, techniques and procedures (TTPs).
What do breach and attack simulations look like?
BAS can encompass a wide range of scenarios and events. It’s often helpful to work through a few examples when trying to describe a new technology. A single BAS module might simulate exfiltrating data using DNS, or scanning an internal subnet for vulnerable Windows systems (perhaps scanning for TCP services on 139 and 445). A common financial breach scenario might involve chaining a few simulations together.
- Privilege escalation on a workstation (emulating a malware infection on a single host)
- Scanning adjacent systems to explore the network and look for potential pivot points
- Further privilege escalation and adjacent scanning
- Command and control (C2) traffic, allowing the attacker to control these malware trojans
- The eventual exfiltration of valuable customer data that could be used for identity theft or payment fraud.
All these simulations can run in sequence over the course of a few seconds or minutes. They could also be run on a regular schedule, though that could lead to potential problems, as we’ll discuss later on.
Use Case: Controls Testing
As previously mentioned, functional testing is one of the core use cases for BAS. It’s common for simple mistakes to be made when implementing security controls, so even the most basic testing can yield effective results. Beyond functional testing, practical testing takes things to the next level, assessing how well controls can prevent or detect attacks currently in use.
The challenge of testing controls against practical attacks is that attackers and defenders are playing a constant game of leapfrog. Defend against one type of attack and techniques are modified or developed to evade these defenses. This is one advantage commercial offerings will likely have over open-source tools—the R&D budget to keep up with current attack techniques.
With the growing popularity of managed security services, BAS can be used to test third-party controls in addition to testing internally-managed controls. In addition, these simulations could be used to assess the maturity of potential acquisitions or subsidiaries.
Use Case: Staff Testing
In most disciplines, we gain experience, maturity, and proficiency through a constant cycle of practice, failure and improvement. In security, we traditionally lack the ability to practice and fail in a safe way, that doesn’t involve bringing down production systems and/or losing our jobs. BAS offers a way to make this possible, as one of its goals is to trigger alerts and incidence without causing damage.
Avoiding risk, however, does limit what BAS is capable of simulating. Still, there is immense value in being able to train staff on realistic scenarios, using production tools. Incident response teams and analysts learn to find alerts and stop attacks in the same systems they’ll use to prevent and investigate real attacks. BAS is useful by itself or in conjunction with tabletop exercises and live red team assessments.
Use Case: Product Testing
How do we know that security products work? In many cases, vendors don’t even provide the ability to functionally test products. The EICAR anti-virus test file is a great example of a functional test and is still very useful for ensuring an AV engine is capable of seeing files or traffic. Where’s the EICAR for the rest of these products? Asking these questions a few years ago resulted in both entertaining and insightful responses.
Questions for the defenders out there. How do you test your anti-APT/breach detection/next-gen kit? How do you validate it works?
— Adrian Sanabria (@sawaba) September 11, 2015
@sawaba a few occasions where I've been contracted to write malware for the specific purpose of testing defense tools. Batting .1000
— Dave Maynor (@Dave_Maynor) September 11, 2015
@sawaba if you can't structure a simple test to verify in under an hour, it's probably some over-complicated yet well-marketed "solution".
— Stephen DiCato (@stephendicato) September 11, 2015
@sawaba We use implants that simulate network traffic of well known APT implants. You'd be surprised what expensive IDS's completely miss it
— chris doman (@chrisdoman) September 11, 2015
Dave Maynor, Stephen DiCato and Chris Doman provide some great suggestions, but many security teams might not have the skills to both recognize the need for test product tests and the ability to create them. The lack of development skills in security teams is a critical skills gap, but security-focused developers are rare and highly sought.
For commercial BAS vendors, however, it’s an opportunity. It’s true that, individually, most of these tests are simple to design and implement. When an organization needs a hundred or more of these tests, however, a commercial offering starts making more sense. For more cash-strapped organizations, the rise in open source BAS projects is a welcome trend.
@sawaba @SwiftOnSecurity see if it catches a pen test or internal red team?
— Gabe The Engineer (@gdbassett) September 11, 2015
There are also issues with using a penetration test or other external consulting resources to test products and controls. It is expensive to do so and because it is expensive, it will be infrequent. Ideally, security teams need the ability to run these tests as frequently as they need to. Process improvement can only occur at the speed of the feedback loop. If this loop iterates only once or twice a year (the pen test approach) the loop is unlikely to ever have a noticeable impact on the quality of the security program.
.@sawaba @SwiftOnSecurity Heh. He he. He he ha ha aHA HAHAHAhaha ha ha...
— design by eddie(tm) (@random_eddie) September 11, 2015
[wipes eyes]
[ahem]
We're strictly a faith-based community.
There’s always a comedian. There’s enough truth to Eddie’s statement for it to strike a nerve, however. Ideally, BAS should be embedded into existing tools and workflows, becoming a ‘sanity’ check for IT and security teams. To be effective, this sanity check needs to be available throughout the product lifecycle:
- When considering a purchase
- Following the first implementation of a product
- After the product’s configuration is changed
- Following any maintenance activities that require the product to be moved, restarted or switched off
- After replacing or decommissioning a product
Use Case: Integration Due Diligence
While the previous use case focuses on testing the product from the customer perspective, this one focuses on a product or services provider ensuring their own work is implemented correctly and is functioning as expected. When an MSSP drops a sensor or other technology into a client’s environment, there should be some level of testing performed. Perhaps even proof should be provided to the customer that everything is working.
When a reseller or integrator installs or rolls out a security product, the final stage should be functional testing at a minimum. For products that require tuning to be effective, functional testing isn’t enough. The product should be tuned using current, practical attack simulations. The customer should require successful testing and a demonstration of efficacy before signing off on the implementation or perhaps even before paying the invoice. Taking things a step further, customers could even require these ‘efficacy demonstrations’ in their contract. Inability to stop or detect common attacks could (and probably should) result in a breach of contract.
The Market
While anyone with even modest skills can simulate attack and breach events, both commercial and open source software exists to automate the process and provide a comprehensive set of pre-built simulations. Python is the ‘language of security’ and most commercial vendors write their simulations in this language.
A common goal in this market is to offer simulations that won’t negatively impact production environments. The idea is that attack simulations should be safe enough to run in production environments on a daily basis. While Metasploit exploits have ratings based on how likely they are to crash services, BAS products are designed to be as close to 100% safe to use within a production environment as possible.
Commercial
Most commercial BAS tools are agent-based and leverage python scripts to run simulations. The advantage of this approach is that agents can act as both attacker and victim, verifying whether or not attacks from one internal host to another were successful or blocked. Python takes us back to a time when commercial security tools didn’t obfuscate their techniques. The ability to read the included python scripts, copy them and add to them is good news for more savvy customers, looking forward to building their own simulations.
Vendors that fit this description include AttackIQ, SafeBreach, and Verodin. Standard in this category are also dashboards that summarize success and failure. Dashboards also show trending statistics, intended to show whether security performance is improving or declining. Threatcare is a BAS vendor that can run almost entirely inside a browser, leveraging javascript to run simulations. Threatcare has an agent also. There are other vendors in this space that sell BAS products alongside services or a larger product portfolio, to be covered in a follow-up piece.
Worryingly, the BAS space has become a bit of a numbers game for marketing. Similar to the heyday of vulnerability scanning vendors, “we’ve got more simulations than the other guy” has become a common metric for comparing performance. Does the customer really need 1500 different ways to exfiltrate data, or is it more important to identify and address as many major gaps in coverage as possible? The answer likely depends on the maturity of the organization.
Ultimately, the commercial BAS space could result in yet another console to be monitored; yet another product to be tuned, yet another security operations role to hire and fill. As previously mentioned, the ideal BAS product integrates into existing workflows as a ‘sanity check’ rather than becoming yet another strain on staff resources.
Free/Open-Source
The open source BAS tools do not resemble their commercial counterparts. Most are designed to be run as standalone tools to test controls within a narrow slice of the security program. That, or they cover more ground but don’t go terribly deep and are designed to be automated and integrated into existing tools.
Guardicore’s Infection Monkey is an example of the former. It takes the form of simulated binary malware and can simulate a variety of attack techniques. It can also spread itself, mimicking many of the same techniques that made the WannaCry and NotPetya malware so destructive.
Uber’s Metta, AlphaSOC’s FlightSIM and Synex Caldera are examples of the latter and represent ‘suites’ of different kinds of attacks and require different levels of effort to get running in given environments. A followup to this piece will explore more open source BAS tools and the pros and cons of each.
What now? Day 1 with BAS versus Day 100
We previously alluded that there could be issues with scheduling regular BAS assessments. On one hand, it makes sense to regularly run simulations to ensure that security controls are working as expected. There have been cases where security appliances didn’t get turned back on after data center maintenance and no one notices for days or weeks. There have been cases where network threat monitoring products have been plugged into a misconfigured switch port.
Just because a positive result is achieved by running a simulation once doesn’t mean it will always be so. The state of systems and the network can change in a way that will later cause security controls to fail, often in unanticipated ways. On the flip side of this, we have to ensure that staff isn’t so used to seeing simulations and green lights on dashboards that they mistake actual attacks for tests.
The value of BAS tools is clear on ‘day 1’ or in specific scenarios—it’s going to help security teams debug their security controls and products. How BAS tools will continue to deliver value and how they will be used on day 100 is less clear. The answer may be that they are ‘point-in-time’ tools, only used when the need arises. Or they might be used daily, checking the same controls and making sure a certain ‘efficacy SLA’ doesn’t drop below acceptable levels. Whatever the answer is, it will likely vary for each organization and care will need to be taken to avoid falling into familiar security product ‘ruts’.
As with most security controls, BAS tools have gaps, so care must be taken to remember that these tools aren’t showing the efficacy of security controls for all infrastructure and probably aren’t testing all controls. If information security had a Hippocratic Oath, it might be interpreted in 2018 as “first, don’t make alert fatigue any worse than it already is.”
Looking to learn even more? MISTI's InfoSec World Conference & Expo provides you with the perfect experience to do so. Here's everything you need to know.
