In today's economy, CISOs are increasingly focused on optimizing and rationalizing their security stacks, while also leveraging automation to reduce the need to hire from the ever-scarce talent supply.
At the same time, many CISOs ask themselves: Which new metrics should we track? Both to justify security budgets to leadership and to drive continuous improvement in their security operations.
Security leaders have typically tracked metrics such as mean time to detect (MTTD) and mean time to respond (MTTR). Of course, these metrics are still important. Detecting and responding quickly to attacks — before they have a material impact on the business — stands as important function of the security team.
But MTTD and MTTR metrics are missing critical information about which attacks were never – and will never be – detected in the first place, either because of important gaps in detection coverage — or due to alerts that got buried in a sea of noisy alerts and were never pursued by the SOC team.
The surprising disparity between perceived and actual coverage
Many organizations are unaware of the disparity between their assumed theoretical security and the defenses they actually have in place.
In fact, according to anonymized and aggregated data from diverse production SIEMs, including Splunk, Microsoft Sentinel, and IBM QRadar – encompassing more than 14,000 log sources, thousands of detection rules, and hundreds of log sources – our analysis found:
- On average, enterprise SIEMs contain detections for fewer than five of the top 14 MITRE ATT&CK techniques employed by adversaries in the wild.
- Fifteen percent of SIEM detection rules are broken and will never fire, primarily because of fields that are not extracted correctly or log sources not sending the required data.
- Only 25% of organizations that forward identity logs to their SIEM, such as Active Directory and Okta, actually use them in detection rules – meaning they’re likely to miss top ATT&CK tactics like Privilege Escalation and Credential Access.
- Seventy-five percent of generic out-of-the-box detection content provided by SIEM vendors is disabled because of noisiness and customization challenges experienced by detection engineering teams.
And according to IDC, 20-30% of all alerts are simply ignored or not investigated in a timely manner, often from classic “alert fatigue” caused by too many noisy alerts.
Measure and validate detection posture to understand the company’s threat exposure
Detection posture metrics should let security teams confidently answer important questions about risk such as “How exposed are we?” in a programmatic manner. And they should base the metrics on standard frameworks like MITRE ATT&CK, which has now become the lingua franca of threat-informed security operations.
Here are some questions that CISOs should ask their teams:
- Are we missing detections for the ATT&CK techniques, adversaries, and crown-jewel assets most relevant to our business?
- Do we have detections that have become noisy, broken, or misconfigured from ongoing changes in our infrastructure – thereby creating additional gaps for attackers to take advantage of?
- Are we missing telemetry for important security monitoring layers like cloud or IAM?
- Can we reduce costs (and complexity) by eliminating underutilized or redundant security tools from our stack?
- What threats are we choosing not to detect based on practicality, cost, and/or the risk profile (that the team should present to the business as part of a formal or informal risk acceptance process)?
- We now have multiple SIEMs (such as Splunk or IBM QRadar plus Microsoft Sentinel or Chronicle SIEM) to optimize costs and monitor hybrid and multi-cloud environments – how do we obtain an aggregate view of the company’s detection coverage?
- Are we proactively developing new detections based on a threat-informed perspective — or simply operating in a reactive, ad-hoc way to the latest requests from internal teams?
Operationalize detection posture management with MITRE ATT&CK
While many SecOps functions are now automated and analytics-driven — such as anomaly detection, alert triage, and incident response — the detection engineering function remains stubbornly dependent on manual processes, tribal knowledge, and individual “SIEM ninjas” that can exit the organization at any time. This reduces agility in responding to constant change in adversary techniques and the company’s attack surface, thereby increasing risk.
Some organizations have now begun to realize the need to gain visibility into their detection posture so they can optimize the effectiveness of their security stack, but they’re doing it with spreadsheets and other time-consuming and error-prone approaches. And they’re taking their best talent and asking them to perform mind-numbing tasks like manually mapping detections to MITRE ATT&CK, rather than allowing them to focus on higher-value activities like threat hunting and researching new and novel attacker techniques.
Thankfully, new approaches have been developed that leverage analytics and automation to assist CISOs and their teams with continuously managing their detection posture.
Measure in business terms
Effective security leadership requires working with the business to support new initiatives, communicate risks, and minimize threat exposure.
With limited resources, it also becomes important to maximize staff efficiency by identifying and prioritizing top use cases that will deliver the most value – and selecting outcome-driven metrics to both justify security budgets and drive continuous SecOps improvement.
Detection Posture Management has emerged as a new and evolving discipline that can help CISOs achieve these objectives in a repeatable and systematic manner.
Michael Mumcuoglu, co-founder and CEO, CardinalOps.
