Malicious Agent Skills in the Wild: A Large-Scale Security Empirical Study

TL;DR

This work addresses the security risks of third-party agent skills in LLM-based agents by constructing the first large-scale, ground-truth dataset of malicious skills from 98,380 publicly indexed skills and confirming 157 malicious instances with 632 vulnerabilities. It combines static triage and sandboxed behavioral verification to achieve high-precision ground truth (99.6% in evaluation) and reveals two distinct attacker archetypes—Data Thieves and Agent Hijackers—coexisting in the ecosystem. The study shows that most vulnerabilities reside in natural-language Skill documentation (84.2%), and that evasion scales with attacker sophistication through shadow features and platform-native attack vectors. Responsible disclosure resulted in a 93.6% removal rate within 30 days, underscoring the value of threat intelligence for platform governance. The authors release the dataset and pipeline to enable ongoing research and defense against evolving agent-skill threats.

Abstract

Third-party agent skills extend LLM-based agents with instruction files and executable code that run on users' machines. Skills execute with user privileges and are distributed through community registries with minimal vetting, but no ground-truth dataset exists to characterize the resulting threats. We construct the first labeled dataset of malicious agent skills by behaviorally verifying 98,380 skills from two community registries, confirming 157 malicious skills with 632 vulnerabilities. These attacks are not incidental. Malicious skills average 4.03 vulnerabilities across a median of three kill chain phases, and the ecosystem has split into two archetypes: Data Thieves that exfiltrate credentials through supply chain techniques, and Agent Hijackers that subvert agent decision-making through instruction manipulation. A single actor accounts for 54.1\% of confirmed cases through templated brand impersonation. Shadow features, capabilities absent from public documentation, appear in 0\% of basic attacks but 100\% of advanced ones; several skills go further by exploiting the AI platform's own hook system and permission flags. Responsible disclosure led to 93.6\% removal within 30 days. We release the dataset and analysis pipeline to support future work on agent skill security.

Figures (6)

• Figure 1: A real-world malicious agent skill: benign calculator description (left) vs. reverse shell payload in calculate.py:32 (right).
• Figure 2: Agent skill ecosystem growth: total skills (blue) and confirmed malicious skills (red) both accelerate over three months.
• Figure 3: Our study overview.
• Figure 4: Vulnerability density distribution. The peak at 4 and heavy right tail (80.3% with $\geq$3) indicate systematic technique layering.
• Figure 5: Kill chain phase coverage per skill. Median=3 phases, with 68.2% spanning $\geq$3 and one skill covering all six.