On February 28, 2026, a Pwn Request vulnerability in Trivy’s GitHub Actions CI/CD workflow handed TeamPCP a privileged personal access token. Aqua Security discovered the breach the next day and initiated credential rotation - but the rotation was incomplete. One unrotated token remained valid for 18 more days. That single oversight was enough to seed a coordinated campaign across five package ecosystems, affecting over 1,000 SaaS environments and an estimated 500,000 machines by April 22.
No zero-days were required. Docker’s post-campaign analysis put it plainly: “This attack did not require any zero-days, novel tradecraft, or nation-state level budgets. The ingredients are stolen credentials and time, and both are abundant right now.”
This post traces the full anatomy of the TeamPCP campaign: the complete timeline across both attack waves, the tag-poisoning and credential-cascade techniques that made it work, detection IOCs for CI/CD pipelines, Kubernetes clusters, and developer hosts, and the hardening measures that would have broken the kill chain at multiple stages. For Kubernetes-specific defense controls (Pod Security Standards, RBAC, NetworkPolicy), see Securing AI/ML Supply Chains on Kubernetes.
What Happened: The Full Timeline
February 28 - The Token That Started Everything
Pwn Requests are a documented GitHub Actions attack pattern. When a workflow uses the pull_request_target trigger and checks out code from the triggering PR, an external contributor can run attacker-controlled code with the repository’s secrets. TeamPCP exploited exactly this in Trivy’s CI/CD on February 27-28, 2026, exfiltrating a personal access token with publisher privileges for the aquasecurity GitHub organization.
Aqua Security discovered the breach on March 1 and rotated credentials. One token was missed. That residual access gave TeamPCP 18 more days to prepare and execute the main attack wave.
graph TD
A["Feb 28: Pwn Request\nTrivy CI compromised, PAT stolen"] --> B["Mar 1: Incomplete rotation\nOne residual token retained"]
B --> C["Mar 19 17:43 UTC\nTrivy v0.69.4 malicious binary released\n76/77 trivy-action tags poisoned"]
C --> D["Mar 19 20:38 UTC\nAttack contained\n~3hr direct exposure"]
C --> E["Mar 20\nnpm worm\n45+ packages"]
C --> F["Mar 22\nDocker Hub images\nTrivy 0.69.5 and 0.69.6"]
C --> G["Mar 23\nCheckmarx VS Code extensions\nKICS GitHub Actions poisoned"]
C --> H["Mar 24\nLiteLLM 1.82.7 and 1.82.8 on PyPI\n~5hr exposure"]
H --> I["Mar 27\nTelnyx 4.87.1 and 4.87.2\nPyPI compromise"]
I --> J["Apr 2-3\nEuropean Commission breach\n340GB exfiltrated, 52,000 emails"]
J --> K["Apr 21-22: Second Wave\nCanisterWorm npm + KICS Docker Hub\nxinference PyPI"]Single PAT theft in February seeded every downstream compromise. The March 19 Trivy attack was the pivot point where harvested credentials from one tool funded attacks against four more ecosystems.
March 19-27 - Five Ecosystems in Nine Days
The main wave opened on March 19 at 17:43 UTC. TeamPCP released Trivy v0.69.4 as a malicious binary and force-pushed 76 of 77 trivy-action tags plus all 7 setup-trivy tags to malicious commits. Aqua detected and contained the attack at 20:38 UTC - approximately 3 hours of direct exposure. Cached pulls in CI runners and pull-through registries extended the effective window considerably.
Every pipeline that ran the poisoned action during that window sent its environment to TeamPCP: cloud provider credentials, npm tokens, PyPI publishing keys, GitHub PATs, and Kubernetes service account tokens. Those harvested credentials funded the next four attacks in sequence:
- March 20: Self-propagating npm worm deployed across 45+ packages using harvested npm publish tokens
- March 22: Malicious Docker Hub images published as Trivy 0.69.5 and 0.69.6
- March 23: Checkmarx VS Code extensions
cx-dev-assist 1.7.0andast-results v2.53compromised; KICS GitHub Actions poisoned - March 24: LiteLLM 1.82.7 and 1.82.8 backdoored on PyPI, approximately 5 hours of active exposure
- March 27: Telnyx 4.87.1 and 4.87.2 compromised on PyPI
April 21-22 - TeamPCP Comes Back
After a 25-day gap, TeamPCP launched a second wave. On April 21, GitGuardian identified CanisterWorm (also reported as CanisterSprawl) - a self-propagating credential stealer deployed to npm that used an Internet Computer Protocol (ICP) canister as its command-and-control endpoint. ICP-hosted infrastructure is harder to take down than a conventional server because the canister runs on a decentralized blockchain network with no single hosting provider to contact.
On April 22 at 12:35 UTC, Docker detected that 7 KICS Docker Hub image tags had been silently overwritten. Xinference PyPI packages were compromised in the same window. TeamPCP then posted on social media: “Thank you OSS distribution for another very successful day at PCP inc.”
How They Did It: No Zero-Days Required
Tag Poisoning: Replacing Trusted Releases with Malware
The primary technique across all waves was tag poisoning: force-pushing existing version tags to new commits containing malicious code while preserving the original tag name. In GitHub Actions, most teams reference actions by tag (aquasecurity/trivy-action@v1). When TeamPCP force-pushed the v1 tag to a malicious commit, every workflow using that tag silently started running attacker-controlled code on its next run.
Metadata cloning made the attack harder to detect. The malicious commits copied original author details and timestamps to appear indistinguishable from legitimate releases in the Actions UI.
The same technique applied to Docker Hub: publishing malicious images under existing tag names (trivy:0.69.5, trivy:0.69.6) so pipelines using those tags would silently pull the backdoored image without any visible indication of the substitution.
Credential Cascade: Each Compromise Funds the Next
graph LR
A["Stolen PAT\nFeb 28"] --> B["Authenticate as\nlegitimate publisher"]
B --> C["Push malicious\npackage or action tag"]
C --> D["CI runner pulls\npackaged code"]
D --> E["Malware harvests\nenv vars, cloud tokens\nSSH keys, K8s service accounts"]
E --> F["Stolen credentials\nfund next target"]
F --> B
E --> G["Exfiltration to C2\nAES-256-CBC encrypted\ntpcp.tar.gz archives"]The attack architecture was self-amplifying. Each compromised tool ran with elevated CI/CD privileges. Harvested credentials from that tool became the access vector for the next target.
The exfiltration mechanism used AES-256-CBC encryption with RSA-4096 OAEP, sending compressed tpcp.tar.gz archives to attacker-controlled C2 endpoints. When primary C2 channels were blocked, fallback exfiltration used stolen GitHub PATs to create repositories named tpcp-docs for staged data transfer.
Persistence Beyond the Package Manager
The LiteLLM compromise introduced the most aggressive persistence mechanism in the campaign. Version 1.82.8 added a litellm_init.pth file to Python’s site-packages directory. Python automatically executes .pth files on every interpreter startup - not just when LiteLLM is imported - meaning the malware activates in any Python process on the system.
The critical detail: pip uninstall litellm does not remove the .pth file. The backdoor survives the uninstall.
On Kubernetes hosts, the credential harvester deployed privileged DaemonSet pods requesting hostPID, hostNetwork, and full host filesystem mounts, enabling complete infrastructure compromise across all cluster nodes. The DaemonSet naming pattern was node-setup-*, chosen to blend in with legitimate Kubernetes infrastructure workloads.
The KICS Docker Hub variant deployed a different persistence path: a systemd backdoor at ~/.config/sysmon/sysmon.py that polled checkmarx.zone every 50 minutes. This backdoor used the user-agent string KICS-Telemetry/2.0 to masquerade as legitimate telemetry traffic.
The Scale Nobody Expected
1,000+ SaaS Environments, 500,000 Machines
Mandiant, as reported via SANS ISC Diary 32864, estimated the campaign affected over 1,000 SaaS environments and approximately 500,000 machines, with hundreds of thousands of stolen secrets in active circulation. Affected sectors included financial services, technology, retail, legal, insurance, and education across the US, Europe, Middle East, South Asia, and Australia.
The downstream impacts reached government infrastructure. The April 2-3 European Commission breach - attributed via the SANS ISC compilation of ShinyHunters and CipherForce reports to the Trivy supply chain entry vector - resulted in 340 GB of data exfiltrated and 52,000 email files exposed, with CERT-EU confirming compromise. Sportradar AG confirmed 26,000 users’ personal information was exposed.
Why 59% of Compromised Machines Were Build Runners
GitGuardian’s 2026 State of Secrets Sprawl report analyzed 6,943 confirmed-compromised systems across the campaign and found that 59% were CI/CD runners, not developer workstations. Build infrastructure accumulates secrets from every job that runs through it: cloud provider keys, container registry credentials, Kubernetes service accounts, database connection strings, and API tokens for every integrated service.
Two statistics explain why incomplete credential rotation cascades the way it did:
- 64% of credentials leaked through GitHub in 2022 were still valid in January 2026, per GitGuardian’s 2026 Secrets Sprawl report. The 2025 edition of the same report found 70% still valid three years after initial exposure.
- The average remediation time for a leaked credential is 94 days. Only 2.6% are revoked within one hour of detection.
Aqua rotated credentials the morning after discovery. One token slipped through. That single oversight, compounded by the 94-day average remediation window, gave TeamPCP 18 days of residual access. The arithmetic of slow remediation is what converted an incomplete rotation into a five-ecosystem campaign.
Are You Affected? Detection Checklist
Network IOCs
Search DNS and proxy logs for connections to any of these indicators:
| Domain/IP | Campaign Stage |
|---|---|
scan.aquasecurtiy[.]org (45.148.10.212) | Trivy primary C2 (typosquatted domain) |
checkmarx[.]zone | Checkmarx C2, persistence polling every 50 min |
models.litellm[.]cloud | LiteLLM exfiltration endpoint |
audit.checkmarx[.]cx | KICS Docker Hub exfiltration |
83.142.209.203 | Additional C2 IP |
tdtqy-oyaaa-aaaae-af2dq-cai.raw.icp0.io | ICP-hosted CanisterWorm C2 |
plug-tab-protective-relay.trycloudflare.com | GitHub Actions exfiltration tunnel |
# Check DNS logs for known C2 domains (adjust log path to your environment)
grep -E "aquasecurtiy\.org|checkmarx\.zone|models\.litellm\.cloud|audit\.checkmarx\.cx" \
/var/log/dns*.log
# Check for the KICS-specific User-Agent in proxy logs
grep "KICS-Telemetry/2.0" /var/log/proxy*.logFile System IOCs
| Path | Indicator |
|---|---|
litellm_init.pth in site-packages | Python startup persistence, survives pip uninstall |
~/.config/sysmon/sysmon.py | Systemd backdoor, polls C2 every 50 minutes |
/tmp/pglog | Staging area for exfiltration data |
tpcp.tar.gz (any path) | Consistent exfil archive name across all stages |
# Search for the .pth persistence file
find / -name "litellm_init.pth" 2>/dev/null
# Inspect all .pth files in Python site-packages directories
python3 -c "import site; print('\n'.join(site.getsitepackages()))" | \
while read dir; do find "$dir" -name "*.pth" -print; done
# Check for systemd backdoor
ls -la ~/.config/sysmon/sysmon.py 2>/dev/null
systemctl --user status sysmon.service 2>/dev/nullKubernetes IOCs
| Indicator | Description |
|---|---|
Pods matching node-setup-* | CanisterWorm DaemonSet naming pattern |
Containers named kamikaze or provisioner | Malicious container names |
DaemonSets in kube-system with hostPID/hostNetwork | Privilege escalation indicators |
# Check for DaemonSet pods using the malware naming pattern
kubectl get pods -A | grep 'node-setup-'
# Audit all DaemonSets in kube-system for unexpected entries
kubectl get daemonsets -n kube-system
# Find pods running with elevated host access
kubectl get pods -A -o json | \
jq '.items[] | select(.spec.hostPID==true or .spec.hostNetwork==true) | .metadata.name'Package Version Audit
These versions are confirmed compromised and should not be running in any environment:
| Package | Compromised Versions |
|---|---|
| Trivy Docker image | 0.69.4, 0.69.5, 0.69.6 |
| trivy-action | All tag-referenced versions (76 of 77 tags were poisoned) |
| LiteLLM (PyPI) | 1.82.7, 1.82.8 |
| Telnyx (PyPI) | 4.87.1, 4.87.2 |
| cx-dev-assist (VS Code) | 1.7.0 |
| ast-results (VS Code) | 2.53 |
graph TD
A{"Did your pipeline use Trivy,\nKICS, or LiteLLM between\nMar 19 and Apr 22?"}
A --> |No| Z["Lower risk\nAudit other packages\nfor unexpected postinstall hooks"]
A --> |Yes| B{"How was the\npackage referenced?"}
B --> |Tag or latest| C["HIGH RISK\nCheck version against\ncompromised list above"]
B --> |Full SHA or digest| D["Verify SHA against\nknown-good release"]
C --> E["Search network logs for C2 domains\naquasecurtiy.org, checkmarx.zone\nmodels.litellm.cloud"]
E --> F{"C2 contact\nfound?"}
F --> |Yes| G["INCIDENT RESPONSE\nRotate all credentials\nRebuild affected runners\nAudit K8s cluster for DaemonSets"]
F --> |No| H["Check file system IOCs\nSearch for litellm_init.pth\nCheck sysmon.service"]
H --> I{"Persistence\nIOCs found?"}
I --> |Yes| G
I --> |No| J["Check K8s cluster\nkubectl get pods -A grep node-setup-\nAudit kube-system DaemonSets"]
J --> K{"Suspicious pods\nor DaemonSets?"}
K --> |Yes| G
K --> |No| L["Likely unaffected\nImplement SHA pinning\nand egress filtering"]Work through this decision tree for each tool in your pipeline. A single C2 contact or persistence IOC warrants full incident response - rotate all credentials found in the affected runner’s environment.
Hardening Your CI/CD Pipeline
The five measures below are ordered by the attack stage they would have blocked in the TeamPCP campaign.
Pin Everything by Digest or SHA
Tags are mutable pointers. Digests and commit SHAs are not. Pinning to an immutable reference is the single change that would have prevented the Trivy action and KICS Docker Hub tag-poisoning attacks entirely.
For GitHub Actions, pin to the full commit SHA:
# Vulnerable: tag can be force-pushed to malicious commit
- uses: aquasecurity/trivy-action@v1
# Safe: force-pushing the tag has no effect on this reference
- uses: aquasecurity/trivy-action@062f2592684a31eb3aa050cc26002765fed9a605 # v0.35.0For Docker images, pin to the content digest:
# Get the verified digest for a trusted image before pinning
docker inspect --format='{{index .RepoDigests 0}}' aquasecurity/trivy:0.68.0
# Dockerfile: mutable tag
FROM aquasecurity/trivy:latest
# Dockerfile: immutable digest - cannot be redirected by tag overwrite
FROM aquasecurity/trivy@sha256:<verified-digest>Audit Lifecycle Scripts Before They Run
npm postinstall hooks execute automatically on npm install. Python .pth files execute on every interpreter startup. TeamPCP used both mechanisms for initial code execution and persistence.
# List all npm packages with postinstall scripts
npm ls --all --json 2>/dev/null | \
jq -r '.. | .scripts?.postinstall? // empty'
# Inspect .pth files in Python site-packages
python3 -c "import site; print('\n'.join(site.getsitepackages()))"
# Review each listed directory for unfamiliar .pth filesAny unexpected postinstall hook or unfamiliar .pth file warrants investigation before the package is allowed to run in production.
Replace Long-Lived Credentials with OIDC
The entire TeamPCP credential cascade depended on long-lived tokens: GitHub PATs, PyPI publishing keys, npm tokens. OIDC (OpenID Connect) eliminates long-lived secrets from GitHub Actions by issuing short-lived tokens scoped to the specific job run.
An OIDC token issued for a job expires when that job ends. A stolen OIDC token has no value for reuse across ecosystems. There is no credential to harvest, rotate, or leak - just a JWT that the cloud provider validates against the GitHub OIDC issuer for the duration of a single build.
Verify Provenance Before You Trust
SLSA (Supply-chain Levels for Software Artifacts) provenance attestations let you verify that a package was built from a specific source repository via a specific CI/CD pipeline. Cosign, from the Sigstore project, lets you verify container image and package signatures against a publisher’s signing key.
Starting from v1.83.0, LiteLLM Docker images are signed with cosign:
# Verify a LiteLLM container image before pulling
cosign verify \
--certificate-identity="https://github.com/BerriAI/litellm/.github/workflows/release.yml@refs/heads/main" \
--certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
ghcr.io/berriai/litellm:main-latestA failed verification would have flagged the malicious LiteLLM images before they ran in any pipeline.
Filter Egress from Build Runners
The LiteLLM malware exfiltrated data to models.litellm.cloud. The KICS backdoor polled checkmarx.zone. The Trivy payload routed exfiltration through a Cloudflare Tunnel. All of these required outbound network access from the CI/CD runner.
Build runners have a predictable set of legitimate outbound destinations: source code hosts, package registries, container registries, and cloud provider APIs. Restricting egress to an explicit allowlist blocks both C2 communication and exfiltration even if malicious code reaches the runner.
graph TD
subgraph prevent["Prevent: Stop Malicious Code from Running"]
A["SHA and Digest Pinning\nBlocks tag-poisoning attacks\nTrivy action, KICS Docker Hub"]
B["Provenance Verification\nCosign and SLSA attestations\nAll package compromises"]
end
subgraph detect["Detect and Contain: Catch Execution Early"]
C["Lifecycle Script Auditing\nnpm postinstall and .pth file monitoring\nCanisterWorm, LiteLLM persistence"]
D["Egress Filtering\nAllowlist for CI outbound traffic\nC2 communication and exfiltration"]
end
subgraph limit["Limit: Reduce Credential Blast Radius"]
E["OIDC Short-Lived Tokens\nNo residual credentials to harvest\nEntire credential cascade chain"]
end
prevent --> detect
detect --> limitEach layer targets a specific stage of the TeamPCP kill chain. SHA pinning eliminates the entry point. Lifecycle auditing and egress filtering catch execution. OIDC tokens eliminate the harvested credentials that funded the downstream cascade.
What This Means for Supply Chain Security
The most important takeaway from TeamPCP is not technical. It is the accessibility of the attack. Docker’s assessment: “The bar for this kind of attack has collapsed. The ingredients are stolen credentials and time, and both are abundant right now.”
Security tools are the highest-value targets in your supply chain because they sit inside the trust boundary by design. Trivy and KICS run as part of your security posture. When they are compromised, the attacker is inside the system you rely on for detection - and they are running with the elevated privileges those tools require to do their job.
The remediation gap data from GitGuardian explains why one incomplete rotation cascaded into a five-ecosystem campaign: 94-day average remediation time, 2.6% of secrets revoked within one hour, 64% of credentials from 2022 still valid in 2026. The math is unforgiving. A single missed token, combined with the industry baseline for credential remediation, produced 18 days of residual access.
The hardening measures above (SHA pinning, lifecycle script auditing, OIDC tokens, provenance verification, egress filtering) are individually straightforward. None require significant tooling investment. The TeamPCP campaign demonstrated at scale what the cost of skipping them has become.
Frequently Asked Questions
How do I check if my CI/CD pipeline pulled a compromised TeamPCP package?
Check for Trivy Docker images tagged 0.69.4, 0.69.5, or 0.69.6 using docker images | grep trivy. Search GitHub Actions workflow files for trivy-action references that use tags instead of full commit SHAs. Check all Python site-packages directories for litellm_init.pth - this file survives pip uninstall litellm. Audit npm packages for unexpected postinstall hooks using npm ls --all --json | jq -r '.. | .scripts?.postinstall? // empty'. Then search DNS and proxy logs for connections to scan.aquasecurtiy.org, checkmarx.zone, models.litellm.cloud, or audit.checkmarx.cx.
Did TeamPCP use any zero-day vulnerabilities?
No. The entire campaign used stolen publisher credentials and legitimate package publishing flows. The initial access was a Pwn Request vulnerability in a GitHub Actions workflow - a documented attack pattern. No novel exploitation was required at any stage. Docker’s post-campaign analysis: “This attack did not require any zero-days, novel tradecraft, or nation-state level budgets. The ingredients are stolen credentials and time.”
Why did TeamPCP target security tools like Trivy and KICS instead of application libraries?
Security scanners run with elevated CI/CD privileges by necessity: cloud credentials, publishing tokens, Kubernetes service account tokens, and repository secrets are all accessible from within a scanner’s execution environment. Compromising a scanner gives attackers everything in the pipeline in a single move. Security tools are also implicitly trusted - malicious updates receive less scrutiny than changes to business logic packages. As Endor Labs noted, TeamPCP strategically targeted security-adjacent tools that run with elevated privileges and are rarely audited with the same rigor applied to application dependencies.
What is digest pinning and how does it prevent tag-poisoning attacks like TeamPCP?
Tags like v1 or latest are mutable pointers that can be redirected to different content at any time. TeamPCP force-pushed 76 of 77 trivy-action tags to malicious commits. Digest pinning uses the immutable SHA-256 hash of the actual content instead of the tag name. For GitHub Actions, pin to the full commit SHA: aquasecurity/trivy-action@062f2592684a31eb3aa050cc26002765fed9a605. For Docker images, pin to the image digest using the sha256: prefix. A tag overwrite has no effect on a digest-pinned reference because the reference names specific content, not a pointer.
How long were the compromised packages available before they were detected?
Detection times varied by package. Trivy v0.69.4 was active for approximately 3 hours (March 19, 17:43 to 20:38 UTC). LiteLLM 1.82.7 and 1.82.8 were active for approximately 5 hours on March 24. KICS Docker Hub images on April 22 were detected within approximately 30 minutes. However, cached copies in CI runners, pull-through registries, and mirrors extended the effective exposure window long after the malicious versions were removed. Docker warned explicitly: “a clean pull won’t remove what’s already been cached.” If your runner cached any of these images during the exposure window, the malicious binary persisted in that cache regardless of registry-side removal.