A Large-Scale Collection Of (Non-)Actionable Static Code Analysis Reports
Dávid Kószó, Tamás Aladics, Rudolf Ferenc, Péter Hegedűs
TL;DR
The paper tackles alert fatigue in static code analysis by building NASCAR, a large, real-world Java dataset that distinguishes actionable from non-actionable warnings. It introduces a data-collection approach based on differential analysis across commit pairs on GitHub, leveraging PMD and SpotBugs to generate warnings and labeling them through code-change context. The resulting public dataset contains over 1.2 million warning instances (1.227M in the full set, deduplicated to 1.083M) with a substantial skew toward non-actionable warnings, and it provides replication tooling and documentation to support ML research aimed at filtering non-actionable alerts. This resource enables more accurate modeling of warning usefulness and supports broader improvements in SCA tool usability for Java developers.
Abstract
Static Code Analysis (SCA) tools, while invaluable for identifying potential coding problems, functional bugs, or vulnerabilities, often generate an overwhelming number of warnings, many of which are non-actionable. This overload of alerts leads to ``alert fatigue'', a phenomenon where developers become desensitized to warnings, potentially overlooking critical issues and ultimately hindering productivity and code quality. Analyzing these warnings and training machine learning models to identify and filter them requires substantial datasets, which are currently scarce, particularly for Java. This scarcity impedes efforts to improve the accuracy and usability of SCA tools and mitigate the effects of alert fatigue. In this paper, we address this gap by introducing a novel methodology for collecting and categorizing SCA warnings, effectively distinguishing actionable from non-actionable ones. We further leverage this methodology to generate a large-scale dataset of over 1 million entries of Java source code warnings, named NASCAR: (Non-)Actionable Static Code Analysis Reports. To facilitate follow-up research in this domain, we make both the dataset and the tools used to generate it publicly available.