The hard sell: which key performance indicators to use in reports

When security pros begin to look for industry-wide standards for reporting metrics, they generally find a simple answer — there isn't one. The metrics that security professionals use to define key performance indicators are usually as unique as the companies for which they work. When framing security needs, goals and accomplishments of an organization, security officers keep the uniqueness of a company in mind, including business goals, risk management strategies and maturity of the security structure, says Khalid Kark, senior analyst at Forrester Research.

"I started off a year and a half ago working on security metrics and trying to figure out if I could narrow down the metrics to ones that everyone can use across the board," says Kark. "And what I quickly found is that there is no uniformity. The metrics depend on the size of a company and the maturity of the security organization. There are lots and lots of variations of what might work well in a particular instance."

"One truth across the board, however, is that companies can't just throw money at the problem. Effective C-level executives are familiar with their own company's risk management strategy — including compliance concerns, assets and types of threats it faces from the outside world and within its own cubicles. Companies using repeatable statistics to show improvement in targeted areas — or the rise of new threats — have superior metrics to those reflecting money spent, says Max Caceras, director of product management at Core Security.

"I would say that there is not a general consensus on what type of metrics to show. Some folks will report operational metrics, which don't necessarily report how security has improved, but just say that this and that were deployed this month," he says. "They don't really mean anything in terms of security. Some organizations are further down the line, and they understand that measurement is the key thing in terms of reporting metrics."

Testing for results

One way to measure the effectiveness of a corporate security structure is to make testing a routine. To ensure security, some executives take a page from the book of military planners by creating and running real-time war-games exercises simulating mass attacks on their networks. No matter what size an organization is, or what industry vertical it resides in, penetration testing is an effective method of reporting what strides a company has made and where additional attention needs to be paid, says Karim Toubba, vice president of marketing and business development at Red Seal.

"Most CSOs say that the only way they can rest at night is to say, ‘I have vulnerabilities and I have issues.' I constantly measure those vulnerabilities and know which risks will be acceptable to the business and what I have to get my team to fix as a priority," says Toubba. "It's a combination of technology and policy. One of the things that we do is to start to give some business context to the vulnerability, for instance, business values for the host."

Many organizations test for easy targets in their system beyond software vulnerabilities and areas of non-compliance. Checking the security awareness of employees is another method used by corporations to check just how secure their infrastructure — including their workforce — really is. Attackers, shown in study after study to be increasingly profit-driven, are certainly mindful that it may be easier to glean valuable information from an employee rather than a network, says Kris Kendall, principal engineer, Mandiant.

"Another thing is to build the security awareness of your people. The low-hanging fruit for a hacker has gone from, ‘I'm going to attack routers and servers,' to ‘I'm going to attack your people,'" Kendall says. "One service that we see a fair amount of is social engineering exercises. You actually get a percentage — 25 percent of your people were willing to give up their password over the phone. Generally, you do phishing and phone calls in those types of exercises. It's super easy to use as a metric."

Measuring compliance

Boardroom executives and CEOs are also well aware of who Sarbanes-Oxley and other compliance regulations were written to influence: themselves. With SOX threatening executives with hefty fines or jail time for non-compliance, corporate officers want to know how much work has to be done to gain compliance, says Malcolm Palmer, director of product management at Cybertrust, which was acquired by Verizon Business in July.

"In terms of specific metrics, the key is whether they have a security plan and are they checking the implementation of that plan," says Palmer. "They're basically checking that baseline set of controls and doing that over an entire enterprise. That's very important because it lets people know how they stand up against the baseline."

Executives also want to ensure that security staffers have an eye on the future when ensuring company compliance, knowing that likely it is only a matter of time until Congress — or legislatures in states where the company operates — pass legislation demanding adherence to new standards. Demonstrable metrics showing adherence to existing control frameworks are a step toward future compliance goals, says Kark.

"I've been talking to a lot of CSOs who are in the process of going through individual compliance programs, or have instituted programs to get compliant, and they realize that tomorrow another regulation is going to come along. They say, ‘Let's not worry as much about individual regulations.

Let's worry about having a good control framework, and let's develop that framework and then worry about regulations to comply with,'" he says.

The key language

When reporting key performance indicators, IT professionals can make the best case for their budgets by placing the statistics in terms that the board pays the most attention to.

Security professionals should be careful to tie their reports to overall company strategy and show exactly how security investment — and the progress provided by it — affects the company's bottom line. Showing where past incidents have cost money can be a helpful measuring stick, says Kendall.

"The best indicators are the number of real incidents that have cost us money, or to say, for example, ‘We had zero incidents last year that cost us money.' So you have to be able to measure the real things first, the number of confirmed incidents, virus outbreaks, how much data we lost. Those are the easiest metrics to measure, but the problem is that the goal is to make sure none of those happen at all," he says.

Making the case

"One of the comments I have about the process is that the metrics are meaningless unless you have well-thought-out goals and stakeholders, and I would always couch my metrics in terms of assets and the policies guarding those assets. But if you don't know what your assets are, you can't protect that," Kendall says.

Put into simple terms, security professionals charged with reporting to the corporate heads are at their best when they speak the board's language — money, says Palmer.

"You really want to deal with direct cost. You want to be able to show X transactions per second and per hour, and attach security to how they allow you to put that into your business plan and the savings that were done by implementing that process," he says.

MEASURING KPIs:
Operational metrics