Learning to Triage Taint Flows Reported by Dynamic Program Analysis in Node.js Packages

TL;DR

This work tackles the challenge of prioritizing vulnerability reports generated by dynamic taint analysis in Node.js packages. It evaluates a range of machine learning approaches—classical ML, graph neural networks, large language models, and hybrid GNN-LLM systems—on a new benchmark, Triage-JS, comprising 1,883 npm packages with taint-flow reports. Fine-tuned LLMs achieve the best overall performance (e.g., $F_1=0.915$), substantially reducing manual triage work while preserving high vulnerability coverage (e.g., $Recall=0.992$ at $Precision=0.8$). Graph-based methods provide competitive baselines, but combining GNNs with LLMs yields limited gains due to high agreement between the modalities. The results demonstrate that ML-enhanced triage can dramatically accelerate real-world vulnerability triage and inform future co-design of program analysis and ML tools.

Abstract

Program analysis tools often produce large volumes of candidate vulnerability reports that require costly manual review, creating a practical challenge: how can security analysts prioritize the reports most likely to be true vulnerabilities? This paper investigates whether machine learning can be applied to prioritizing vulnerabilities reported by program analysis tools. We focus on Node.js packages and collect a benchmark of 1,883 Node.js packages, each containing one reported ACE or ACI vulnerability. We evaluate a variety of machine learning approaches, including classical models, graph neural networks (GNNs), large language models (LLMs), and hybrid models that combine GNN and LLMs, trained on data based on a dynamic program analysis tool's output. The top LLM achieves $F_{1} {=} 0.915$, while the best GNN and classical ML models reaching $F_{1} {=} 0.904$. At a less than 7% false-negative rate, the leading model eliminates 66.9% of benign packages from manual review, taking around 60 ms per package. If the best model is tuned to operate at a precision level of 0.8 (i.e., allowing 20% false positives amongst all warnings), our approach can detect 99.2% of exploitable taint flows while missing only 0.8%, demonstrating strong potential for real-world vulnerability triage.

Figures (8)

• Figure 1: All triage approaches evaluated
• Figure 2: Node.js attacker model
• Figure 3: Provenance graph generated from the toygrep package. Upper-left section contains source code (Listing \ref{['lst:toy_code']}); Bottom-left section contains the driver program (Listing \ref{['lst:driver_program']}); Right section presents the provenance graph. Ovals are node. Black edges are untainted, red edges are tainted flows.
• Figure 4: Precision of the top N% outputs for GNN, XGBoost (average pooling) and full fine-tuning of DeepSeek-R1-Distill-Qwen-7B. The shaded area represents the standard deviation across five random seeds.
• Figure 5: Cohen's Kappa coefficient for the predictions of GNN, DeepSeek-R1-Distill-Qwen-7B, and XGBoost on the test split. The Kappa values are computed across all combinations of random seeds for each method. The higher the Kappa value, the more agreement there is between the two methods.