NOTE: This story was updated after publication following the release of additional details by CircleCI on Jan.5, 2023
CircleCI, a developer tool provider with over one million users, is urging customers to immediately rotate all secrets following a breach of the company’s systems.
The DevOps company said in an advisory published late Wednesday that it is currently investigating a security incident, with the wording suggesting that the breach happened on Dec. 21, 2022 and remained unnoticed over the holiday period.
“We wanted to make you aware that we are currently investigating a security incident, and that our investigation is ongoing,” CircleCI chief technology officer Rob Zuber noted in the advisory. “At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well.”
Specifically, the company strongly recommends that users rotate “any and all secrets” stored in CircleCI, including those stored in project environment variables or contexts. It also advises users to “review internal logs for any unauthorized access starting from Dec. 21, 2022, through Jan. 4, 2023.”
In addition, for projects using API tokens, CircleCI said it has invalidated those, and users will need to replace them.
According to CircleCI’s newest update on Jan.5, the company highlighted that it has eliminated the risk that led to the incident and provided detailed instructions on how users could rotate the secrets.
Additionally, the company suggested that users to further increase pipeline security by using OIDC tokens wherever possible to avoid long-lived credentials in CircleCI. Users are also advised to take advantage of IP ranges to limit inbound connections to the systems to known IP addresses or adopt Contexts to activate the sharing of environment variables across projects to rotate secrets via API automatically.
CircleCI hasn’t released details about how many customers may be affected and what systems were exposed in the latest advisory, and SC Media has reached out to them for further information.
Security researcher Daniel Hückmann said in a Jan.4 Twitter post that he was investigating the use of a Thinkst Canary AWS token that was improperly accessed on Dec.27 before Wednesday’s advisory. Such tokens are deployed throughout IT networks and function as tripwires that can alert organizations to otherwise imperceptible intrusions or malicious activity.
Hückmann also shared an attacker IP address for AWS credentials stored in CircleCI: 54.145.167.181. “Search cloudtrail logs for events from this IP. I expect there are other indicators, but this one is high fidelity,” he noted.
SC Media has reached out to Hückmann for further details.
While CircleCI claimed that no unauthorized actors are active in the systems, the potential impact of the breach can be notable as the company has a large technology supplier base, including PyTorch, HashiCorp, and Snyk and others as customers.
In addition, Leonid Belkind, co-founder and chief technology officer at Torq, told SC Media that it can be challenging for users to rotate all secrets manually and quickly.
“Rotating a secret means disabling the credential on the original system it belongs to, then issuing a new credential with exactly the same permissions to keep the pipeline running and updating it in CircleCI. Identifying who in the organization can and should perform these operations for each credential and then ensuring they are all done is a tremendous undertaking,” Belkind explained.
Indeed, earlier today, a CircleCI user expressed concern over the difficulty of rotating the secrets in CircleCI’s online discussion forum while seeking solutions and was told by an employee that there are currently no company-generated scripts available to facilitate mass rotation of secrets across a large number of projects.
The breach comes four months after GitHub warned that a phishing campaign has been using fake CircleCI email notifications to steal GitHub account credentials and two-factor authentication codes.
Earlier in 2019, CircleCI was hit by a data breach after the compromise of a third-party vendor, which led to the compromise of user data, including usernames, email addresses, IP addresses, and organization names.
