Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CHASE: LLM Agents for Dissecting Malicious PyPI Packages

Takaaki Toda, Tatsuya Mori

TL;DR

This work tackles the challenge of reliably detecting malicious PyPI packages amid sophisticated multi-stage malware and the reliability issues of LLM-based analysis. It introduces CHASE, a Plan-and-Execute, Supervisor-Worker multi-agent architecture that pairs specialized workers (e.g., Deobfuscator, Web Researcher) with deterministic tools and a minimal, tiered LM stack to maintain stateful, reliable analysis. On a real-world dataset of 3,000 packages, CHASE achieves 98.4% recall with 0.08% false positives and a practical median analysis time around 4.5 minutes per package, with retries improving coverage for timeouts. The authors also validate report usefulness via cybersecurity professionals, and propose CHASE as a scalable blueprint for reliable AI-powered security tools capable of handling growing software supply chain complexity.

Abstract

Modern software package registries like PyPI have become critical infrastructure for software development, but are increasingly exploited by threat actors distributing malicious packages with sophisticated multi-stage attack chains. While Large Language Models (LLMs) offer promising capabilities for automated code analysis, their application to security-critical malware detection faces fundamental challenges, including hallucination and context confusion, which can lead to missed detections or false alarms. We present CHASE (Collaborative Hierarchical Agents for Security Exploration), a high-reliability multi-agent architecture that addresses these limitations through a Plan-and-Execute coordination model, specialized Worker Agents focused on specific analysis aspects, and integration with deterministic security tools for critical operations. Our key insight is that reliability in LLM-based security analysis emerges not from improving individual model capabilities but from architecting systems that compensate for LLM weaknesses while leveraging their semantic understanding strengths. Evaluation on a dataset of 3,000 packages (500 malicious, 2,500 benign) demonstrates that CHASE achieves 98.4% recall with only 0.08% false positive rate, while maintaining a practical median analysis time of 4.5 minutes per package, making it suitable for operational deployment in automated package screening. Furthermore, we conducted a survey with cybersecurity professionals to evaluate the generated analysis reports, identifying their key strengths and areas for improvement. This work provides a blueprint for building reliable AI-powered security tools that can scale with the growing complexity of modern software supply chains. Our project page is available at https://t0d4.github.io/CHASE-AIware25/

CHASE: LLM Agents for Dissecting Malicious PyPI Packages

TL;DR

This work tackles the challenge of reliably detecting malicious PyPI packages amid sophisticated multi-stage malware and the reliability issues of LLM-based analysis. It introduces CHASE, a Plan-and-Execute, Supervisor-Worker multi-agent architecture that pairs specialized workers (e.g., Deobfuscator, Web Researcher) with deterministic tools and a minimal, tiered LM stack to maintain stateful, reliable analysis. On a real-world dataset of 3,000 packages, CHASE achieves 98.4% recall with 0.08% false positives and a practical median analysis time around 4.5 minutes per package, with retries improving coverage for timeouts. The authors also validate report usefulness via cybersecurity professionals, and propose CHASE as a scalable blueprint for reliable AI-powered security tools capable of handling growing software supply chain complexity.

Abstract

Modern software package registries like PyPI have become critical infrastructure for software development, but are increasingly exploited by threat actors distributing malicious packages with sophisticated multi-stage attack chains. While Large Language Models (LLMs) offer promising capabilities for automated code analysis, their application to security-critical malware detection faces fundamental challenges, including hallucination and context confusion, which can lead to missed detections or false alarms. We present CHASE (Collaborative Hierarchical Agents for Security Exploration), a high-reliability multi-agent architecture that addresses these limitations through a Plan-and-Execute coordination model, specialized Worker Agents focused on specific analysis aspects, and integration with deterministic security tools for critical operations. Our key insight is that reliability in LLM-based security analysis emerges not from improving individual model capabilities but from architecting systems that compensate for LLM weaknesses while leveraging their semantic understanding strengths. Evaluation on a dataset of 3,000 packages (500 malicious, 2,500 benign) demonstrates that CHASE achieves 98.4% recall with only 0.08% false positive rate, while maintaining a practical median analysis time of 4.5 minutes per package, making it suitable for operational deployment in automated package screening. Furthermore, we conducted a survey with cybersecurity professionals to evaluate the generated analysis reports, identifying their key strengths and areas for improvement. This work provides a blueprint for building reliable AI-powered security tools that can scale with the growing complexity of modern software supply chains. Our project page is available at https://t0d4.github.io/CHASE-AIware25/
Paper Structure (29 sections, 3 figures, 7 tables)

This paper contains 29 sections, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. The PyPI Ecosystem and the Malicious Package Threat
  4. Anatomy of a Malicious Package
  5. Current Detection Approaches and Their Limitations
  6. Agentic AI for Complex Analysis
  7. CHASE Architecture
  8. Overall Design Philosophy
  9. System Components
  10. The Supervisor Agent
  11. Worker Agents and Their Specializations
  12. Reliability-Oriented Agent Coordination
  13. State-Based Communication for Context Integrity
  14. Specialized Agents and Adaptive Planning to Prevent Failures
  15. Implementation
  16. ...and 14 more sections

Figures (3)

  • Figure 1: The multi-agent architecture of CHASE, using a Plan-and-Execute workflow and a Supervisor-Worker model.
  • Figure 2: Design Principle for Reliable LLM-Tool Interaction: (a) A fine-grained tool design forces the LLM to handle high-entropy strings (e.g., hash values, shown in red), which is a hallucination-prone task. (b) Our composite tool, inspect_domain_or_url_using_virustotal, encapsulates this entire workflow, shielding the LLM from volatile data handling to enhance overall system reliability.
  • Figure 3: The analysis trace for libstrreplacecpu-7.3 as generated by CHASE.