TEAMPCP’S RAMPAGE CONTINUES
The data was scraped and exfiltrated from 2,500 users of a compromised AI package.
Terabytes worth of credentials, many belonging to the world’s biggest and most sensitive organizations, have been exposed in a supply-chain attack on LiteLLM, an open source tool that streamlines AI-driven software development. Microsoft, Amazon, Cisco, Samsung, and Salesforce are only a handful of the entities whose access secrets were exposed.
The revelation was posted on Tuesday and Wednesday by security firms CloudSEK and Hudson Rock. CloudSEK said it found cloud keys, repository tokens, SSH keys, Kubernetes secrets, package publishing credentials, environment variables, and AI provider keys that could allow attackers to gain access to more than 2,500 organizations.
40 minutes is all it takes
The credentials were extracted during a 40-minute window in March while the victims used compromised versions of LiteLLM downloaded from the package’s official location in the Python Package Index repository. Hudson Rock said it made the discovery after analyzing a 195TB file that it obtained. Neither firm identified the source of the information.
The LiteLLM compromise was the result of a previous supply-chain attack that infected the widely used vulnerability scanner Trivy. Other software infected in the campaign includes KICS and the Telnyx Python SDK. TeamPCP, a ramshackle but extremely capable gang largely made up of teenagers, took credit for the attack, and researchers have largely corroborated the claim.
“I’ve confirmed the data is legit by the way, multiple victim orgs,” independent security researcher Kevin Beaumont said. “It contains a significant volume of sensitive content at orgs. It’s a massive supply chain breach due to poor AI security—not because AI is the threat, but teens can run circles around orgs obsessed with rushing out AI and poor DevOps security.”
The compromised versions of all four software packages contained code that accessed the memory of infected machines, scraped its contents, and exfiltrated it through an attacker-controlled channel. The data is filled with an assortment of information. Interspersed in the wall of data are credentials to software pipelines maintained by the tens of thousands of organizations that ran LiteLLM during the 40-minute span that the supply-chain attack remained active.
In all, both security firms said some 434,000 CI/CD (continuous integration/continuous delivery) software pipelines had credentials exposed after running the compromised LiteLLM versions. In many cases, researchers at CloudSEK and Hudson Rock had trouble identifying the organizations the credentials belonged to. For instance, an email address in the dump from the domain @siriusxm.com ultimately didn’t indicate a breach at the satellite broadcaster, but rather one within the infrastructure of SiriusXM subsidiary AdsWizz.
A trove of internal corporate secrets, exposing sensitive tokens for platforms such as Salesforce (SALESFORCE_CLIENT_SECRET), Slack (SLACK_SIGNING_SECRET), and Microsoft Azure environments.
Credit: Hudson Rock
A trove of internal corporate secrets, exposing sensitive tokens for platforms such as Salesforce (SALESFORCE_CLIENT_SECRET), Slack (SLACK_SIGNING_SECRET), and Microsoft Azure environments. Credit: Hudson Rock
A full list of organizations is here. The researchers had high confidence that these organizations had their credentials exposed:
- Nvidia Corporation
- Amazon Web Services (AWS)
- Samsung Electronics
- samsung.com
- Salesforce, Inc.
- Cisco Systems, Inc.
- F. Hoffmann-La Roche AG
- ServiceNow
- Siemens AG
- S&P Global
- Airbus US Space & Defense
- John Deere
- Regeneron Pharmaceuticals, Inc.
- London Stock Exchange Group (LSEG)
- Thomson Reuters
- FedEx
- Munich Remunichre.com
- MediaTek Inc.
- Volkswagen AG
- Deloitte
- The Kroger Co.
- Siemens Energy
- Thales Group
- X Corp (Twitter)
- Zscaler, Inc.
- Epic Games
- Orange S.A.
- HP Inc.
- Philips
- Fortum Oyj
- Vodafone Group Plc
- Carl Zeiss AG
- Deutsche Bahn AG
- NGINX, Inc.
- BT Group
- Liebherr
- Krungthai Bank Public Company Limited
- Roku, Inc.
“Many CI/CD pipelines are configured generically,” Hudson Rock said. “The dumped variables contain active database passwords, third-party API keys, and cloud credentials without any identifiable company email, custom domain string, or internal server name. This means countless organizations currently have active secrets sitting in this database, completely unaware of their exposure.”
Welcome to the new world of supply-chain attacks
Both firms are urging all organizations that used the compromised versions of LiteLLM—particularly those listed in the high-confidence section of the list—to thoroughly rotate all credentials in their pipelines. Hudson Rock instructed any organization that uses any AI proxy infrastructure, third-party CI/CD vulnerability scanners, or downstream AI packages to immediately audit their environment for versions 1.82.7 and 1.82.8 of LiteLLM, the two compromised versions of the software.
The firm advised all those affected to perform “aggressive credential revocation,” assume any secret accessible to the LiteLLM environment is compromised, invalidate and rotate all cloud keys, Kubernetes service account tokens, and GitLab/GitHub PATs, and audit logging and egress filtering.
As a cautionary tale, CloudSEK said that Trivy developers rotated, but failed to fully revoke an automation token over a 20-day window. The lapse gave the attackers a nearly three-week period to force-push malicious code to third-party builds that used the vulnerability scanner. As Beaumont observed, organizations’ rush to integrate AI into their software delivery systems has also greatly contributed to the scale of the damage.
Update:There are already signs that some of the affected organizations aren’t taking the disclosure with the seriousness warranted. After this post went live, Beaumont reported:
These creds date from about March. One of the orgs impacted told me they’d rotated them all and it’s a nothingburger, so I looked at their responsible disclosure policy, it allows trying creds, so I tried them all. Almost every one worked. Submitted report. One of the biggest US techcos.
Ultimately, the new revelations concerning the LiteLLM supply-chain attack underscore the growing threat of such campaigns and hence the importance of maintaining vigilance around the use of open source software that, when infected, can spread rapidly across the Internet.
“The key takeaway is how supply chains have evolved to make a single upstream breach affect thousands of companies simultaneously,” Alon Gal, co-founder and chief technology officer of Hudson Rock, wrote in an email. “A window of roughly 40 minutes in which the LiteLLM dependency was hacked led to over 430,000 instances in which millions of secrets were harvested. This magnitude pushes us into a completely new world regarding the type of response required from the cybersecurity industry.”
Post updated to add image.
Dan Goodin is Senior Security Editor at Ars Technica, where he oversees coverage of malware, computer espionage, botnets, hardware hacking, encryption, and passwords. In his spare time, he enjoys gardening, cooking, and following the independent music scene. Dan is based in San Francisco. Follow him at here on Mastodon and here on Bluesky. Contact him on Signal at DanArs.82.
