Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilities

TL;DR

This work investigates package hallucination as a security risk in AI-assisted software development. By employing the garak framework, it measures how often LLMs hallucinate non-existent package imports across Python, JavaScript, and Rust, varying model size and coding specialization. The study reveals that larger models tend to hallucinate less, JavaScript is somewhat safer on average, and there is a strong inverse relationship between coding benchmark performance and hallucination propensity, especially with HumanEval. It also distinguishes induced from natural hallucinations and discusses practical mitigations, such as pre-cutoff package verification, to secure dependency workflows in real-world development environments. Overall, these findings highlight the need for multi-objective optimization in model development and concrete defenses to curb supply-chain risks in AI-assisted coding.

Abstract

Large Language Models (LLMs) have become an essential tool in the programmer's toolkit, but their tendency to hallucinate code can be used by malicious actors to introduce vulnerabilities to broad swathes of the software supply chain. In this work, we analyze package hallucination behaviour in LLMs across popular programming languages examining both existing package references and fictional dependencies. By analyzing this package hallucination behaviour we find potential attacks and suggest defensive strategies to defend against these attacks. We discover that package hallucination rate is predicated not only on model choice, but also programming language, model size, and specificity of the coding task request. The Pareto optimality boundary between code generation performance and package hallucination is sparsely populated, suggesting that coding models are not being optimized for secure code. Additionally, we find an inverse correlation between package hallucination rate and the HumanEval coding benchmark, offering a heuristic for evaluating the propensity of a model to hallucinate packages. Our metrics, findings and analyses provide a base for future models, securing AI-assisted software development workflows against package supply chain attacks.

Figures (5)

• Figure 1: Package Hallucination Rate by language, averaged across all models
• Figure 2: Distribution of package hallucination rate split by code-specialized and non-code models. Lower y-axis scores are better. Coding-specific models plots are solid-shaded, other models are hatch-shaded. Note that for every language, coding-specific models exhibit higher propensity to hallucinate packages.
• Figure 3: Model Size (x) vs PHR (y). Lower PHR is desirable. Small and low-hallucination models are in the bottom left. Pearson product-moment correlation coeff. of size vs. PHR is $-0.541, p=0.00114$; coeff of ln(size) vs. PHR is $-0.593, p=0.00028$.
• Figure 4: MBPP Coding score vs. package hallucination rate
• Figure 5: HumanEval Coding score vs. package hallucination rate