Business teams have widely adopted data lakes as a way to consolidate information from multiple sources and improve business results with analytics and automation. Security leaders have taken note, and are increasingly aligning their strategy to the rest of the enterprise. In the race to stay ahead of determined and sophisticated adversaries, defenders are now adopting the security data lake model to gain visibility, reduce busywork and empower stakeholders across the company to do their part in avoiding the next breach.
The growth in cloud, connected devices and remote work has only increased the volume, variety and complexity of security data. Traditional security tools operate largely on their own data and place the burden of connecting the dots on overstretched security teams. Even solutions that were designed to consolidate logs across multiple sources end up creating silos when the cost of comprehensive ingest and retention is prohibitively high — making it hard for security teams to achieve necessary prevention, detection and response outcomes.
Advances in cloud data platforms, however, have made it easier and more cost-effective to consolidate data at scale. Security teams that recognize this have turned their company’s central data platform into a home for security datasets of all sizes. Terabytes and petabytes of log data from sources such as firewalls, endpoint agents and cloud infrastructure are unified with this approach.
Beyond storage, modern data platforms support fast analytics with languages such as SQL and Python to transform this unified data into actionable insights. As a result, less time is wasted on false positives, and investigating potential breaches is becoming less manual, slow and error prone. In a positive cycle of improvement, leading security teams can now easily add new datasets from across IT and the rest of the business that provide more context, further reducing false positives and busy work.
Security data lakes, as implemented on a modern cloud platform, are ideal for machine learning and other forms of advanced analytics. With security data housed in an organization’s primary cloud platform, security and analytics teams can work side by side to solve problems and build solutions, applying the latest techniques to better detect anomalies and automate processes. In this way, the CISO fully aligns with the CIO, working from the same advanced data stack.
The benefits aren’t limited to threat detection, either. Risk and compliance, identity and access, and vulnerability management are all better served when data is consolidated to share and access. And leveraging existing BI tools enables generating reports that can show leadership how cyber posture is improving, or encourage stakeholders in other departments to act.
Innovative security vendors understand these benefits and support the security team owning its data on its company’s existing data platform. For example, Hunters, Panther Labs and Securonix now offer solutions in a “connected application” model where they plug into the customer’s security data lake. I expect this model to become the standard for security solutions and data platforms that share an open and data-driven approach.
Unifying security data and maintaining its availability without limitations is essential to turning the tide in the battle against cybercriminals. As the SolarWinds hack showed, responders must be equipped to investigate over a year into the past. In response to this and other breaches, the U.S. government has said it will require federal agencies to expand their event retention and deploy behavior analytics to help mitigate future cyberattacks. This standard should apply to all security conscious organizations. As the need for handling security data grows, cloud-based security data lakes have become the most effective and cost-efficient way to apply sophisticated analytics on the ever-expanding data volumes. In the epic battle to secure the enterprise, the good guys have a new way to carry the day.
By Omer Singer, Head of Cybersecurity Strategy, Snowflake
