Security platforms like Security Information and Event Management can be useful tools when it comes to staying ahead of the threat landscape. That is, if you have the money, staff and expertise to make it work.
Those who don’t often struggle to get the most out of their SIEMs, or triage and investigate the daily deluge of security alerts they can produce. One network security analyst SC Media spoke with at Black Hat said he had been managing a SIEM for 15 years and this particular problem is so bad, he believes there are only between 50-100 companies that have the staffing to manage the level of alerts and the know-how to configure and tune their platforms to limit false positives.
He’s not the only one complaining. In a survey conducted by SumoLogic last year, 70 percent of IT security officials with direct responsibility for corporate security said their volume of alerts had doubled in the past five years and 83% said they are experiencing “alert fatigue.” Amazingly, 99% reported that high volumes of alerts caused problems for their IT security team, and an identical number are hoping for better built-in automation capabilities to pare that volume down.
It illustrates the limitations these platforms have in their current form, not just for the large swath of organizations who can’t afford them, but even for many who do. One company was frustrated enough they decided to try something new: building their own.
Speaking at the Black Hat cybersecurity conference in Las Vegas, executives from Two Sigma Investments, a technology-based hedge fund, laid out their journey from SIEM customer to creator. Ethan Christ, their vice president and head of the security identity, monitoring and response team, said the company originally invested a million dollars in a commercial SIEM license, but it had a number of drawbacks. There was a hard cap — one terabyte — on the amount of data it could ingest on a daily basis.
“Routine overages of this cap would lead to the loss of our ability to search our data and we’re going to have to be constantly mindful and diligent of our daily investment now,” Christ said, describing their circumstances. “Ideally, we want[ed] a system that wouldn’t have to restrict us, but does give us the flexibility to collect more without paying exponentially more.”
Next, the company wanted something that could be tuned and customized to meet their specific threat intelligence needs, get quicker detections for potentially suspicious activity around high value assets, wouldn’t force them to “shoehorn” their data into predefined fields, or deal with interoperability concerns and secure their sensitive data from malicious hackers, third-party suppliers and insider threats. Finally, they wanted to get the most bang for their buck when it came to logging.
“Reliability was another key requirement. We rely heavily on logs for our forensic capabilities, so ensuring their accuracy and completeness was critical,” Christ continued.
Some of these features could be added or incorporated to their existing SIEM, but often with higher costs around storage, server hardware and operational overhead. As it was, their million-dollar investment couldn’t do what they bought it to do: analyze large swaths of data in a convenient and reliable way.
These drawbacks all pointed Two Sigma Investments to the conclusion that it would be more effective to just build their platform, one that lived in the cloud, could handle the volume of data they needed to process and were built specifically to understand their technology environment, what was normal and what should raise red flags.
This required a substantial investment, but Christ said their internal assessments indicating that upgrading their existing on-premise SIEM to perform all these capabilities would have cost several million dollars more in annual licensing fees, as well as additional costs around maintenance, software updates and support.
The project ultimately required 6,000 lines of code and the equivalent of approximately half an FTE year to get “minimum viable product” and another three months to work through use cases around logging. The end result gave them the ability to ingest up to 5 petabytes of data every day and a dramatic drop in false positives. While it wasn’t cheap, it also saved money. Two Sigma Investments calculated that they saved at least $3.5 million in annual licensing costs and another $600,000 in annual maintenance contracts.
Even with their positive experience, the company recognizes that this approach is a tough lift for many organizations.
“I want to specifically call this out, that our choice to roll our own SIEM was heavily impacted by decisions our business had made, and that may certainly not be the case for everyone,” Christ said.
Still, for those who can and don’t want to rely on commercial software or wait for the Great Automation Revolution to finally arrive, it could offer another potential option for getting your SIEM on the right path.
