Many Hands Make Light Work: An LLM-based Multi-Agent System for Detecting Malicious PyPI Packages

TL;DR

LAMPS presents a modular, LLM-driven multi-agent system for detecting malicious PyPI packages by allocating distinct roles to specialized agents for retrieval, extraction, per-file classification, and conservative package-level verdicts. By integrating a fine-tuned CodeBERT classifier with LLaMA-3–based reasoning within the CrewAI framework, it achieves superior accuracy and robustness compared with MPHunter, TF-IDF stacking, and RAG baselines on two datasets, with statistically significant improvements. The approach emphasizes auditability, reproducibility, and extensibility, addressing context-window limitations and obfuscation challenges that hinder monolithic detectors. This work demonstrates the feasibility and benefits of distributed LLM reasoning in software supply chain security and points to future expansion to other ecosystems and the inclusion of dynamic analysis for broader coverage.

Abstract

Malicious code in open-source repositories such as PyPI poses a growing threat to software supply chains. Traditional rule-based tools often overlook the semantic patterns in source code that are crucial for identifying adversarial components. Large language models (LLMs) show promise for software analysis, yet their use in interpretable and modular security pipelines remains limited. This paper presents LAMPS, a multi-agent system that employs collaborative LLMs to detect malicious PyPI packages. The system consists of four role-specific agents for package retrieval, file extraction, classification, and verdict aggregation, coordinated through the CrewAI framework. A prototype combines a fine-tuned CodeBERT model for classification with LLaMA-3 agents for contextual reasoning. LAMPS has been evaluated on two complementary datasets: D1, a balanced collection of 6,000 setup.py files, and D2, a realistic multi-file dataset with 1,296 files and natural class imbalance. On D1, LAMPS achieves 97.7% accuracy, surpassing MPHunter--one of the state-of-the-art approaches. On D2, it reaches 99.5% accuracy and 99.5% balanced accuracy, outperforming RAG-based approaches and fine-tuned single-agent baselines. McNemar's test confirmed these improvements as highly significant. The results demonstrate the feasibility of distributed LLM reasoning for malicious code detection and highlight the benefits of modular multi-agent designs in software supply chain security.

Figures (9)

• Figure 1: Motivating example of supply-chain risk in PyPI. An adversarial upload can embed obfuscated logic in setup.py or auxiliary modules; developers unknowingly install the package, and downstream applications are affected at runtime. The example motivates semantics-aware analysis and conservative package-level aggregation.
• Figure 2: Architecture of LAMPS showing agent responsibilities and communication flow.
• Figure 3: Figure 3: Illustrative flow of how a single file-level prediction is processed in LAMPS. The Classifier Agent (using CodeBERT) produces a binary label, and the Verdict Agent (using LLaMA-3) generates a human-readable rationale in JSON format.
• Figure 4: Overview of the datasets. (a) D$_1$: balanced distribution of benign and malicious setup.py files. (b) D$_2$: imbalanced distribution of benign and malicious files across full packages. (c) Distribution of files per package in D$_2$ (log scale), showing most packages are small with a long tail of larger ones. (d) Distribution of file lengths in D$_2$ (tokens), illustrating variability and heavy tails.
• Figure 5: Confusion matrices for MPHunter, TF-IDF + Stacking and LAMPS on D$_1$. Compared to MPHunter, and TF- IDF +Stacking baselines, LAMPS significantly reduces misclassifications, achieving a better balance between benign and malicious detection.