A JetBrains TeamCity critical vulnerability is being actively exploited to create hundreds of administrator accounts on more than 1,400 servers.
JetBrains TeamCity is a continuous integration and continuous development (CI/CD) platform used to automate the building, testing and deployment of software projects. On Monday, JetBrains patched and disclosed a critical flaw, tracked as CVE-2024-27198, which enables authentication bypass to gain administrative control over TeamCity servers.
LeakIX reported Wednesday that 1,442 on-premises instances of TeamCity showed “clear signs of rogue user creation.” LeakIX is a search engine that indexes internet-connect services and provides information about vulnerable services and data leaks.
The TeamCity vulnerability was discovered by Rapid7, which said in its disclosure published Monday that exploiting the bug would give an attacker “full control over all TeamCity projects, builds, agents and artifacts,” setting the stage for potential supply chain attacks.
“We started hearing from some of our customers who were noticing their servers had been compromised shortly after Rapid7 made their full disclosure with exploit examples, and the subsequent exploit module,” JetBrains TeamCity Solutions Engineer Daniel Gallo told SC Media. “The first was just hours after this information was published online.”
All managers of on-premises TeamCity instances must upgrade to version 2023.11.4 to prevent exploitation of CVE-2024-27198. LeakIX stated any users who have been running a vulnerable version should “assume compromise.”
TeamCity users should monitor servers for unusual activity
Gallo provided SC Media with several recommendations for users who may have patched late or suspect a compromise. Firstly, users should take the server offline to isolate it from external and internal networks, Gallo said.
The server’s “Audit” screen should be checked for actions such as “unknown user access tokens created or deleted, any unknown users created, login actions, etc.,” which can be filtered by the “User actions” category, Gallo explained.
Attackers have been creating between 3 and 300 new users accounts on compromised servers, with a pattern of using eight alphanumeric characters as the user name, LeakIX told BleepingComputer.
Additional avenues to check for suspicious activity include the “teamcity-activities” log file, where users check for unknown plugins, projects or build configurations, and event logs on the host operating system, which should be checked for unknown events logged by external processes, Gallo said.
“Checks should be carried out for the presence of any unrecognized processes running in the background on the host server, such as malware, cryptomining, and other unauthorized tools,” said Gallo.
If suspicious activity is detected, the server should remain isolated as credentials are rotated for all users.
“There will also be a varying number of credentials for any external services that TeamCity and its host operating system had access to, such as repositories and connections (e.g. Slack, AWS, Docker Registries, etc), the TeamCity database connection, and network drives,” Gallo noted. “If the server has been compromised, any build agents connected to the server during that period should also be considered compromised.”
Stephen Fewer, principal security researcher at Rapid7, told SC Media any server with signs of compromise should be reviewed by an incident response team to uncover the full extent of the breach.
“The best practices would be to rebuild that server, including the underlying operating system, so any persistence the attacker achieved would no longer prevail,” Fewer said.
LeakIX said a total of 1,711 vulnerable TeamCity servers were detected as of Wednesday, down from the 2,000 it discovered on Monday, according to an X post.
The Shadowserver Foundation, a non-profit that tracks vulnerabilities and malicious activity online, said on X that it first detected exploitation of CVE-2024-27198 on March 4 at 22:00 UTC. Shadowserver’s time series dashboard for the vulnerability currently shows 1,182 vulnerable servers as of Wednesday.
Cybersecurity company GreyNoise has observed 33 malicious IPs targeting CVE-2024-27198 for exploitation, with a large spike of activity detected on Tuesday, March 5.
JetBrains, Rapid7 clash over disclosure timing
Both JetBrains and Rapid7 published timelines of events leading up to the disclosure of the critical authentication bypass flaw, along with the high-severity path traversal flaw tracked as CVE-2024-27199.
While the timelines mostly agree with regard to the dates of contacts and what was generally discussed, they reveal an apparent clash between the disclosure philosophies of the two parties.
JetBrains first accepted Rapid7’s report on Feb. 19, with Rapid7’s timeline stating they first reached out on Feb. 15 and followed up four days later.
On Feb. 21, JetBrains reserved the CVEs for the flaws and suggested publicly disclosing them a few days after releasing a fix, while emailing customers about the patch in the interim. JetBrains said this approach aligns with its Coordinated Disclosure Policy.
Rapid7 replied that this approach conflicted with its own Vulnerability Disclosure Policy, as they considered the delay in public disclosure to be “silent patching.” Following additional discussion about disclosure policies and Rapid7’s definition of silent patching, JetBrains said it decided on Feb. 23 not to coordinate disclosure with Rapid7.
JetBrains released the fixed version of TeamCity on March 4 with an initial release that did not disclose the security flaws and did not notify Rapid7 before publishing the release. The company began emailing customers about the patch and published its blog post disclosing the security vulnerabilities about an hour after the initial release.
Minutes after this second blog post, according to JetBrains, Rapid7 contacted JetBrains saying they would publish their disclosure within a few hours. Rapid7’s timeline suggests they contacted JetBrains before the publication of the second blog post saying they planned to publish their version within 24 hours in accordance with their policy on silent patching.
Rapid7’s blog post about the vulnerabilities was ultimately published about five and a half hours after the patch was released.
“The reason we ultimately disagreed with the coordinated disclosure with Rapid7 was due to their insistence on publishing full exploit details at the same time as a fix being made available to our customers. This would immediately provide attackers with the necessary tools to exploit TeamCity servers without any opportunity for our customers to apply mitigation measures,” Gallo told SC Media.
“We fully support the timely disclosure of vulnerability details when a fix is released. However, we provide only the necessary details for customers to understand the vulnerability’s scope and severity (such as the CVE details and description of the attack vector), enabling them to take the appropriate actions without disclosing so much information that it facilitates straightforward exploitation,” Gallo added.
Rapid7 declined to provide a comment to SC Media in response to JetBrains’ criticism of its disclosure timing.
However, the company’s previous blog post on silent patching, which it cites in its timeline, states that while delaying the public disclosure of vulnerability details prevents a “casual attacker” or “script kiddie” from misusing them, “you’re also excluding a huge population of IT protectors.”
“Most significantly, you’re excluding the most important audience for the patch: the regular IT administrators and managers who need to sort out the incoming flow of patches based on some risk and severity criteria and make the call for downtime and update scheduling based on that criteria,” the 2022 post, authored by then-Director of Research Tod Beardsley, stated.
