Three lessons for DevOps from the CircleCI breach

For development teams that are still struggling to dig out from the mess caused by the hack of CircleCI in December, recent days brought some good news, and a warning about the security risks posed by vulnerable developer accounts and development pipelines.

First the good news: writing on January 13th, about one week since CircleCI first disclosed the security breach, CTO Rob Zuber reported that the company observed the theft (exfiltration) of data on just a single day of the incident and that just five of the company’s estimated 1,700 customers subsequently informed CircleCI of unauthorized access to third-party systems as a result of the incident. The news suggests that the attack on CircleCI was targeted and limited in scope, not a wholesale assault on CircleCI’s customers and their data.

Now for the bad news: the rest of Zuber’s post made clear that the breach might not end with CircleCI. And it put DevOps organizations on notice about the risks posed by vulnerable developer accounts – risks that many DevOps organizations are poorly equipped to address. It might take months or years to fully understand the impact of the CircleCI attack, but here are some of the important lessons of the incident that we can already grasp, and that DevOps organizations should not overlook. 

According to Zuber’s account of events, CircleCI was first compromised on December 16, 2022, when an “unauthorized third party” used unnamed malware to gain access to a CircleCI engineer’s laptop and steal a valid, 2FA-backed single sign on (SSO) session cookie. The theft of the valid session cookie let the attackers impersonate the targeted employee in a remote session and gain access to CircleCI production systems using the employee’s privileges, Zuber said.

The first link in the chain of the CircleCI attack was a simple one: endpoint protection. Because the anti-malware software running on the developer’s workstation failed to flag the malicious code, attackers gained access to critical assets (like the session cookie) needed to facilitate their attack.

Anti-malware might seem like an afterthought in the context of sophisticated supply chain attacks, but almost every attack of note starts with the compromise of a single endpoint. And that makes endpoint protection software the point of the spear in defending organizations from attacks. That means development organizations should shore up their malware detection and endpoint protection capabilities as soon as possible.

The CircleCI attack stands as one more argument for implementing user least privilege tools, policies and processes that can limit lateral movement by malicious actors. Zuber’s account pointed out that attackers leveraged the hacked developer’s privileges to access and exfiltrate data from CircleCI databases and stores, including customer environment variables, tokens, and keys.

As part of its response to the incident, CircleCI said it has changed its access policies: restricting access to production environments to “a very limited number of employees.” The company also strengthened authentication, adding additional “step-up authentication” steps and controls that it claims will prevent account hijacking even in the case of a stolen 2FA-backed SSO session. CircleCI also said it was stepping up monitoring and detection of developer behaviors to spot suspicious activity of the kind that characterized the recent breach.

There’s a clear lesson here: DevOps organizations need to invest more time, energy and technology in hardening developer accounts and development environments to compromises.

The use of two-factor authentication to secure access to developer accounts and development systems provides a significant advantage over simple usernames and passwords. However, as the CircleCI breach shows, it's not foolproof nor immune to manipulation. It’s often impossible to prevent attacks. But limiting developer privileges using a “least privilege” framework, coupled with greater emphasis on strong, multi-factor authentication and behavior monitoring can stifle attackers’ ability to move laterally within compromised organizations and escalate their attack.

DevOps organizations need to do more to monitor their exposure to the risk of upstream software supply chain compromises, and that’s one of the main lessons from the CircleCI breach.  CircleCI’s January 4th alert about the compromise amounted to a fire drill for DevOps organizations that had not been closely managing and monitoring the secrets they stored in code, such as API and OAuth tokens or SSH keys. According to CircleCI, that announcement led to a flurry of inquiries from customers about how to identify stored secrets in their code. Subsequently, the company issued a free script to help identify secrets as well as updated documentation and guidance on how to rotate secrets across different platforms.

The CircleCI breach should have DevOps organizations looking closely at their development infrastructure and the policies enacted to manage and rotate secrets. Limiting the access and shortening the useful life of stored secrets offers a great way to limit the impact of any breach.

More broadly, organizations need to expand their supply chain risk analysis from software artifacts to the tools and platforms that make up their development pipeline. As CircleCI noted in its post: “because this incident involved the exfiltration of keys and tokens for third-party systems, there’s no way for us to know if your secrets were used for unauthorized access to those third-party systems.”

With weeks of access to CircleCI’s customer accounts, and with a rich ecosystem of development tools and platforms that integrate with CircleCI, the blast radius of the breach may be much larger. 

For development organizations that have had secrets in a cloud-based CI system exposed, how can the team know that they are building what they’re supposed to build? Can the team rule out an attacker abusing their access to the CI system to push malware straight to production? The sobering truth: one malicious commit to a CI/CD system can quickly cascade from source to production. This can happen in minutes.

Unfortunately, the secrets that were exposed in the CircleCI breach are the kinds of information that can facilitate such supply chain attacks. That’s why DevOps organizations need to increase their scrutiny of software artifacts: putting security controls in place that can verify the integrity of a build before it’s pushed to production. Absent such controls, there’s no assurance that security teams can contain incidents like the CircleCI breach and then detect and prevent software supply chain attacks.

Tomislav Peričin, chief software architect, ReversingLabs