Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Malicious Or Not: Adding Repository Context to Agent Skill Classification

Florian Holzbauer, David Schmidt, Gabriel Gegenhuber, Sebastian Schrittwieser, Johanna Ullrich

Abstract

Agent skills extend local AI agents, such as Claude Code or Open Claw, with additional functionality, and their popularity has led to the emergence of dedicated skill marketplaces, similar to app stores for mobile applications. Simultaneously, automated skill scanners were introduced, analyzing the skill description available in SKILL.md, to verify their benign behavior. The results for individual market places mark up to 46.8% of skills as malicious. In this paper, we present the largest empirical security analysis of the AI agent skill ecosystem, questioning this high classification of malicious skills. Therefore, we collect 238,180 unique skills from three major distribution platforms and GitHub to systematically analyze their type and behavior. This approach substantially reduces the number of skills flagged as non-benign by security scanners to only 0.52% which remain in malicious flagged repositories. Consequently, out methodology substantially reduces false positives and provides a more robust view of the ecosystem's current risk surface. Beyond that, we extend the security analysis from the mere investigation of the skill description to a comparison of its congruence with the GitHub repository the skill is embedded in, providing additional context. Furthermore, our analysis also uncovers several, by now undocumented real-world attack vectors, namely hijacking skills hosted on abandoned GitHub repositories.

Malicious Or Not: Adding Repository Context to Agent Skill Classification

Abstract

Agent skills extend local AI agents, such as Claude Code or Open Claw, with additional functionality, and their popularity has led to the emergence of dedicated skill marketplaces, similar to app stores for mobile applications. Simultaneously, automated skill scanners were introduced, analyzing the skill description available in SKILL.md, to verify their benign behavior. The results for individual market places mark up to 46.8% of skills as malicious. In this paper, we present the largest empirical security analysis of the AI agent skill ecosystem, questioning this high classification of malicious skills. Therefore, we collect 238,180 unique skills from three major distribution platforms and GitHub to systematically analyze their type and behavior. This approach substantially reduces the number of skills flagged as non-benign by security scanners to only 0.52% which remain in malicious flagged repositories. Consequently, out methodology substantially reduces false positives and provides a more robust view of the ecosystem's current risk surface. Beyond that, we extend the security analysis from the mere investigation of the skill description to a comparison of its congruence with the GitHub repository the skill is embedded in, providing additional context. Furthermore, our analysis also uncovers several, by now undocumented real-world attack vectors, namely hijacking skills hosted on abandoned GitHub repositories.
Paper Structure (42 sections, 10 figures, 3 tables)

This paper contains 42 sections, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Agent and Skills.
  4. Skill Marketplaces.
  5. Security on Marketplaces.
  6. Empirical Studies on the Skill Ecosystem.
  7. Security of AI Agents.
  8. Scientific Literature.
  9. Methodology
  10. Cross-Platform Skill Collection
  11. Skills Beyond Marketplaces.
  12. Static Skill Analysis.
  13. Malicious Classification
  14. Platform-based Scanners.
  15. Cisco Skill Scanner.
  16. ...and 27 more sections

Figures (10)

  • Figure 1: Overview of our repository-aware skill analysis approach to reduce the high rate of malicious claims. The approach is tested in a three-stage pipeline encompassing cross-platform skill collection, malicious classification, and repository context analysis.
  • Figure 2: All investigated platforms show an increase in number of weekly added agent skills over time.
  • Figure 3: Overlap of skills published to the marketplaces.
  • Figure 4: Conditional scanner agreement on Skills.sh common skills, shown as $P(B\text{ flags}\mid A\text{ flags})$.
  • Figure 5: Number of Skills.sh common skills flagged by exactly $k\in\{1, \dots,5\}$ scanners.
  • ...and 5 more figures