The development community considers the trend towards shifting security left and incorporating security scans into the build phase throughout the continuous integration/continuous delivery (CI/CD) pipeline an incredibly positive one. It helps organizations speed up their development while catching many security issues before they reach production.
But it will take more than shifting left. It catches most potential security vulnerabilities, but we still can’t ignore the vulnerabilities that slip into production.
We need to expand security, not merely shift to the left. We aim for full lifecycle security that spans the entire build-run-respond loop and includes security forensics into any suspected security incident, also known as secure DevOps or DevSecOps.
Some realities of shifting left
In our recent container security report based on actual Sysdig customer data, we found that 74 percent of customers are scanning container images during the build stage. They are shifting security left as the phrase gets used today. But there’s a concerning trend here: 58 percent of containers run as root. The shift left isn’t enough to prevent the majority of containers from running with excessive privileges.
As a developer, I understand the desire to run containers as root. It’s easier. There’s also a misunderstanding among many developers — they assume that the container can’t do damage to anything beyond the container, even with full privileges. That just isn’t true. Linux kernels are very complex, there’s no way for us to fully understand all the ways that a privileged container could cause havoc in other parts of the system.
Shifting left does not prevent containers from running with root privileges. But there are some other tasks it can’t help with, too. Zero-day vulnerabilities, which have not yet been reported to CVE databases, will slip through even the best image scans in the CI/CD pipeline. Strictly following security best practices during the build phase also can’t necessarily prevent privilege escalation in the production environment. Shifting left does nothing to help organizations understand the cause and extent of any suspected security incident after the fact or to use information from suspected incidents to further improve security.
Cover the entire lifecycle
The industry doesn’t need to just shift left but rather transform security. We can’t consider container security a point-in-time checkpoint that an application goes through on the road to production. It should operate as a continuous, full-lifecycle process that starts as code gets written and does not end until the application comes to end-of-life.
Each part of the application lifecycle has unique security needs. During the build phase, checking the container images against CVE databases, ensuring secure configurations and following access control best practices means that the application remains as secure as possible when it goes into production.
All of those tasks remain important at runtime, but runtime security also requires monitoring container behavior. We will not catch zero-day vulnerabilities in a CVE scan, and sophisticated attackers keep their own databases of unpublished vulnerabilities to exploit. You have to quickly block or kill containers that are behaving suspiciously, regardless of whether or not they have a known vulnerability.
Don’t forget the tools
Even if the teams shifts left and monitors applications from the beginning of the build cycle, it can’t ignore the tools it uses to monitor and manage the environment, both open source and closed-source. These tools have broad access to the company’s codebase and infrastructure and can contain vulnerabilities and backdoors that can compromise the computing environment.
Open source tools are less likely to have unknown security vulnerabilities because they are widely scrutinized by the community. More people from diverse companies applying open source tools in different ways, makes for a more robust review. Shifting left will not improve the security posture of third-party tools and platforms used — but monitoring runtime behavior can alert the team to possible problems and allow it to respond quickly.
Investigate breaches
A “perfect” security posture does not exist anymore. There’s always some level of vulnerability, and organizations need to integrate forensics into their security strategy. Think of this as shifting — or expanding — all the way to the right.
Preparing for the possibility of breaches and taking a proactive approach to minimize damage and learn from security failures has become just as important as integrating security at the build stage. Having a realistic view of security based on minimized risk, fast responses and learning from security failures will ultimately improve the organization’s security posture.
While no organization has a perfect security posture, organizations tend to experience fewer breaches when they follow security best practices and have full-lifecycle security coverage that’s comprehensive and includes shifting security left.
Loris Degioanni, founder and CTO, Sysdig
