An Empirical Study of False Negatives and Positives of Static Code Analyzers From the Perspective of Historical Issues

TL;DR

This paper examines false negatives and false positives in Java static code analyzers from a historical-issues perspective, analyzing 350 developer-confirmed and fixed FN/FP issues across PMD, SpotBugs, and SonarQube to uncover root causes and input characteristics. It provides a taxonomy of root causes, including flawed rule specifications, inconsistent implementations, unhandled language features and libraries, missing cases, IR mishandling, and analysis-module limitations, along with ten input characteristics that trigger issues. A metamorphic testing approach using equivalent input program mutations demonstrates practical utility by uncovering 12 new FN/FP issues (11 confirmed, 9 fixed) and reveals weaknesses in analysis modules through manual rule inspection. The study offers actionable guidance for tool developers, including modular rule design, improved feature coverage, and targeted testing strategies, and it releases all artifacts publicly to support future research and tooling improvements.

Abstract

Static code analyzers are widely used to help find program flaws. However, in practice the effectiveness and usability of such analyzers is affected by the problems of false negatives (FNs) and false positives (FPs). This paper aims to investigate the FNs and FPs of such analyzers from a new perspective, i.e., examining the historical issues of FNs and FPs of these analyzers reported by the maintainers, users and researchers in their issue repositories -- each of these issues manifested as a FN or FP of these analyzers in the history and has already been confirmed and fixed by the analyzers' developers. To this end, we conduct the first systematic study on a broad range of 350 historical issues of FNs/FPs from three popular static code analyzers (i.e., PMD, SpotBugs, and SonarQube). All these issues have been confirmed and fixed by the developers. We investigated these issues' root causes and the characteristics of the corresponding issue-triggering programs. It reveals several new interesting findings and implications on mitigating FNs and FPs. Furthermore, guided by some findings of our study, we designed a metamorphic testing strategy to find FNs and FPs. This strategy successfully found 14 new issues of FNs/FPs, 11 of which have been confirmed and 9 have already been fixed by the developers. Our further manual investigation of the studied analyzers revealed one rule specification issue and additional four FNs/FPs due to the weaknesses of the implemented static analysis. We have made all the artifacts (datasets and tools) publicly available at https://zenodo.org/doi/10.5281/zenodo.11525129.

Figures (6)

• Figure 1: Typical workflow and architecture of a static code analyzer.
• Figure 2: Workflow of our study.
• Figure 3: Illustrative examples for explaining root causes (the code snippets are simplified).
• Figure 4: Input characteristics leading to FNs/FPs.
• Figure 5: Illustrative examples for explaining input characteristics (the code snippets are simplified)