Paradoxically we measure our security testing tools by the noise they generate and then, naturally, we choose the loudest tool. After all, in evaluating vendor A’s tool that generated 500 security-related findings and vendor B’s tool generated just 50 security-related findings, is it possible to trust the quality of the latter? Probably not.
How did we fall into this cycle? First, we never know the expected result. So to begin with, we don’t really know the “target” number of findings the tools need to discover. Maybe then 500 findings does seem to feel more “complete” and make more sense. With no widely accepted benchmark to use when measuring the effectiveness of a cyber security testing tool, the industry defaulted to a measurement that’s applicable across the board: the number of findings.
Second, we always prefer more false positives than one false negative. The fear of missing one critical issue leads us to compromise and suffer sifting through the noise.
Here’s another problem that we created as an industry: duplicate findings. Since we’re incentivizing security vendors to provide us with tools that generate more findings (what we have come to value), tools are now expanding outside their core areas of expertise just to demonstrate more value by offer more findings. We’re at a stage where there’s a 20% finding overlap between tools addressing different use cases. Just take a look at these examples:
- XDR solutions: While their original intention was to prevent or detect a malicious attack, they have evolved to test for vulnerabilities – not the use case that drove the XDR purchase in the first place.
- Web application testing tools: These tools started expanding into external attack surface management – a separate product category altogether.
- Cloud Security Posture Management tools: Testing for cloud misconfigurations has now become a part of any testing tool that scans cloud workloads, not just security testing tools.
Security practitioners Fear of Missing Out (FOMO) and testing vendor market expansions means we’re now in a race for findings, findings, findings: the output of the tools. That push to find more risks distracts us from the ultimate goal security teams have: reducing risk, the outcome of the program. We certainly need findings to reduce risk. But when the measurement and output are all around findings, we forget the reason for purchasing the tool: fixing.
Undoubtedly, we have a broken process. The good news: by understanding what’s driving our behavior we can, as an industry, recognize it and work towards outcome-focused measurements, ones focused on fixes. Given our existing state of affairs – a growing volume of findings and duplication (overlapping coverage), we have no choice but to focus on solutions rather than problems.
We need to transform findings (output) into remediation actions (outcomes). This means shifting our mindset from measuring the number of issues discovered to the number of remediation actions and deployed mitigations. That way, we neutralize the volume and duplication issues and focus on reducing risk not counting risks.
That’s our vision. As an industry, we’re still entangled in our broken processes, trying to keep up with fixing an enormous volume of findings, from separate tools, many times repeated findings in different formats. We must shift to an outcome-focused approach.
Ravid Circus, co-founder and CPO, Seemplicity
