It’s always good to have your radar up on April Fool’s Day, constantly on the lookout for potential pranks or tomfoolery. For one company, what they discovered on April 1 was far from a joke.
Yesterday, software company Codecov, which sells a tool that lets developers measure the testing coverage of their codebase, disclosed that it suffered a breach. In particular, the attackers exploited a bug in the company’s Docker image creation process to gain access to a Bash Uploader script designed to map out development environments and report back to the company. This small modification quietly called out for user credentials that could have been used to access and exfiltrate data from their users’ continuous integration environment.
In a note posted on the Codecov website, CEO Jerrod Engelberg said that any credentials, authentication tokens or keys that were run through an affected customer’s CI process were exposed, and with them the attacker would have had access to any corresponding services, datastores, application code and git repositories that could be accessed by those credentials.
After discovering the breach on April 1, a follow up investigation determined that the threat actor had been in their network since at least January 31, going undetected for months. The vulnerability also affected three other bash uploaders: Codecov CircleCI Orb, Codecov-actions uploader for GitHub and the Codecov Bitrise Step.
“We strongly recommend affected users immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders,” Engelberg advised.
Codecov did not disclose how many of its clients were impacted, only saying they had notified all affected parties in writing. The known details of the intrusion, the nature of the company's work and its customer base has given rise to concerns that the breach could be just the first shoe to drop in a larger software supply chain compromise with potential for messy downstream effects. It lists a number of high-profile customers on its website, including The Washington Post, Atlassian, Mozilla, SweetGreen, GoDaddy and others.
Experts in software development and security reached by SC Media said that the potential for downstream impact on Codecov’s users could be high, but the scope of the damage will depend on a number of factors, such as the identify and motivations of the actor, how Codecov architects their network and what precautions, configurations and access policies each individual user set up for their code environment.
Knowing the identity of the group behind the attack would help shed light on their possible goals, but several observers said the length of time the attackers spent in Codecov’s network and the focus on credentials indicate that they were more interested in getting access to their customers’ code than the company itself.
Unlike SolarWinds and Microsoft, Codecov is not a publicly traded company, has a few dozen employees on staff and measures its annual revenue in the low millions of dollars per year. Despite the high profile of some of their customers, they’ve only existed since 2014 and are not particularly well-known, indicating that the threat actor may have done a fair bit of homework before selecting them as a target.
“I would be leaning [towards espionage] just as a gut inclination. Codecov is off the beaten path,” said John Bambenek, founder of cybersecurity consulting firm Bambenek Labs. “Effectively the compromise involved inserting one line of code and it’s giving credentials. Now there are criminal networks that sell access to organizations and credentials, so it’s not implausible that it’s a fairly sophisticated financial actor that wants to sell them, but if I had to bet, I’m putting my money on espionage.”
The type of credentials, and the access they provide, also matter. Bambenek said if they only got their hands on testing credentials, the impact would be far more limited than if the threat actor had access to credentials for customers’ software production environment.
The extent of Codecov’s network segmentation could also determine in part what customer information and data the group could have accessed. John Zanni, CEO of Acronis SCS, which focuses on data protection and backup recovery services for the public sector, said his company has four separate networks: one for work only devices, one for BYOD home devices, another for guests and family members and one for their software developers that not even the CEO can access.
They also don’t let their developers pull and use open-source code straight from the internet. Before any software is updated, the changes have to go through a code checking review and signing process by another party, something that can guard against both unintentional oversights and insider threats.
“It seems like every time I hire a new developer, that’s the first thing they do with the code they right, so we have to put automated checks in there so the moment somebody tries to do that, they get caught and it stops,” said Zanni.
Robust code signing policies were cited as a best practice by others as well. John Loucaides, vice president of research and development at vulnerability research company Eclypsium said the breach represented a “huge ROI for attackers to attack the supply chain” and that any changes to software code have to be vetted by other parties before approval.
Quinn Wilton, senior researcher at Synopsis Software Integrity, said the breach demonstrates how “code signing is more important than ever, and that transparency around the storage and disposal of those code signing keys is going to be a vital step toward building trust in the channels we all use to distribute software.”
While the attackers went undetected for months, Bambenek said that for a small company with limited resources like Codecov finding, investigating and disclosing a trivial change in their code within three months is actually impressive. He compared it to the SolarWinds breach, where the company itself and multiple customers and federal agencies with larger budgets missed far more substantial code changes in the Orion software build platform for at least a year, if not longer.
“The foothold happened Jan. 31. For an early-stage company like that, that’s solid work,” said Bambenek, who often advises smaller companies on cybersecurity strategy and risk. “Yeah, we’d all like it to be less, but startups are an easy target and so far, it looks like they’re responding to it as well as they can. If they in fact have [only a few dozen] employees, it would surprise me if they have more than one security person.”
