CircleCI’s chief technology officer said malicious hackers infected one of their engineer’s laptops and stole elevated account privileges to breach the company’s systems and data late last year.
In an incident report published late Friday, CTO Ron Zuber said evidence of the breach, first disclosed Jan. 4, 2023, dates back to at least Dec. 16, 2022, when an unauthorized actor compromised the laptop and stole a set of privileged, two-factor authentication-backed credentials.
“Our investigation indicates that the malware was able to execute session cookie theft, enabling them to impersonate the targeted employee in a remote location and then escalate access to a subset of our production systems,” Zuber wrote.
The attacker used that access to pilfer data from “a subset of databases and stores, including customer environment variables, tokens and keys.” Having stolen an unspecified amount of data and evading detection by the company’s antivirus software, the actor shifted to broader reconnaissance activities on Dec. 19, before exfiltrating another batch of data on Dec. 22, including encryption keys needed to decrypt what was taken.
Dan Lorenc, founder and CEO of Chainguard, which bills itself as a software development platform with native supply chain security built in, said the extensive access developers must have to both internal systems as well as production environments make it difficult for endpoint detection systems to spot when they may be acting maliciously.
"These are difficult to detect because devs typically have the most production access but also require the most access to their local systems to do their jobs, rendering most endpoint protection software useless," Lorenc wrote on Twitter.
While it appears only one employee’s account was compromised, Zuber stressed that the breach represented “a system-wide failure” and should not be placed at the feet of any individual.
He said that while the company is now confident that it has closed the attack vector used in the initial compromise and the actor no longer has access to internal CircleCI systems, they cannot guarantee that the stolen information wasn't used to possibly compromise customer systems. Thus far, they aware of “less than five” customers who have reported unauthorized access to their third party systems following the breach.
“If you stored secrets on our platform during this time period, assume they have been accessed and take the recommended mitigation steps,” wrote Zuber. “We recommend you investigate for suspicious activity in your system starting on December 16, 2022 and ending on the date you completed your secrets rotation after our disclosure on January 4, 2023. Anything entered into the system after January 5, 2023 can be considered secure.”
Zuber said the company was first tipped off about suspicious GitHub OAuth activity from one of their customers on Dec. 29. A day later, they had determined an unauthorized party had gained access, spurring a deeper investigation.
“While we are confident in the findings of our internal investigation, we have engaged third-party cyber security specialists to assist in our investigation and to validate our findings,” Zuber wrote. “Our findings to date are based on analyses of our authentication, network, and monitoring tools, as well as system logs and log analyses provided by our partners.”
In response to the discovery, Zuber said CircleCI shut down the employee’s access, and restricted access to production environments to an “extremely small group” of employees to maintain operations, revoked all project and personal API tokens and rotated all GitHub OAuth tokens. He also said the company intends to learn from the breach and has taken a number of other steps to improve their security processes.
They also reached out other third parties who have cloud or SaaS applications that integrate with CircleCI and could have been affected by the compromise, including GitHub, AWS, Google Cloud and Microsoft Azure. As SC Media previously reported, researchers at Mitiga warned that the nature of CircleCI’s platform and its integration with a customer’s cloud environment meant that a compromise of one could easily facilitate the compromise of the other.
“While using the Circle platform, you integrate the platform with other SaaS and Cloud providers your company uses. For each integration, you need to provide the CircleCI platform with authentication tokens and secrets,” wrote Mitiga researchers Doron Karmi, Deror Czudnowski, Airel Szarf and Or Aspir earlier this week. “When a security incident involves your CircleCI platform, not only is your CircleCI platform in danger, [so are] all other SaaS platforms and cloud providers integrated with CircleCI...since their secrets are stored within the CircleCI platform and can be used by a threat actor to expand their foothold.”
CircleCI disclosed earlier this week that they were partnering with AWS to rotate any potentially impacted tokens, and the newest update reveals that they also worked with software development provider Atlassian to rotate BitBucket tokens.
The report provides a list of IP addresses, known VPN and data center providers and other indicators of compromise associated with the threat actor.
