Large Language Models Cannot Reliably Detect Vulnerabilities in JavaScript: The First Systematic Benchmark and Evaluation

TL;DR

This work addresses the unreliability of large language models in detecting JavaScript vulnerabilities by introducing a principled benchmark framework. It presents ForgeJS for automatic data generation, ArenaJS for broad, realistic vulnerability benchmarks, and JudgeJS for automated, reproducible evaluation across complete projects and localization tasks. The large-scale evaluation across seven commercial LLMs reveals persistent gaps in reasoning, robustness to obfuscation/noise, and poor performance under deployment-like false-positive constraints, establishing that reliable LLM-based vulnerability detection remains an open challenge. The authors argue for a methodology that emphasizes comprehensive CWE coverage, avoidance of underestimation, and avoidance of overestimation, and offer a reproducible testbed to drive progress toward deployable solutions for JavaScript security. The work also discusses generalization and future directions, including hybrid analyses and broader language support.

Abstract

Researchers have proposed numerous methods to detect vulnerabilities in JavaScript, especially those assisted by Large Language Models (LLMs). However, the actual capability of LLMs in JavaScript vulnerability detection remains questionable, necessitating systematic evaluation and comprehensive benchmarks. Unfortunately, existing benchmarks suffer from three critical limitations: (1) incomplete coverage, such as covering a limited subset of CWE types; (2) underestimation of LLM capabilities caused by unreasonable ground truth labeling; and (3) overestimation due to unrealistic cases such as using isolated vulnerable files rather than complete projects. In this paper, we introduce, for the first time, three principles for constructing a benchmark for JavaScript vulnerability detection that directly address these limitations: (1) comprehensiveness, (2) no underestimation, and (3) no overestimation. Guided by these principles, we propose FORGEJS, the first automatic benchmark generation framework for evaluating LLMs' capability in JavaScript vulnerability detection. Then, we use FORGEJS to construct ARENAJS-the first systematic benchmark for LLM-based JavaScript vulnerability detection-and further propose JUDGEJS, an automatic evaluation framework. We conduct the first systematic evaluation of LLMs for JavaScript vulnerability detection, leveraging JUDGEJS to assess seven popular commercial LLMs on ARENAJS. The results show that LLMs not only exhibit limited reasoning capabilities, but also suffer from severe robustness defects, indicating that reliable JavaScript vulnerability detection with LLMs remains an open challenge.

Figures (3)

• Figure 1: Overall architecture of the SecJS Framework.
• Figure 2: [RQ1] Performance of seven LLMs on ArenaJS. “Full” = full split; “DN” = denoised split. All scores are percentages.
• Figure 3: [RQ2] Ablation Study Results. All scores are presented in percentage.