Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry

TL;DR

The paper tackles the problem of high false positive rates in enterprise static analysis tools and the associated manual review burden. It presents the first comprehensive industrial evaluation at Tencent, using 433 alarms from an enterprise SAT (BkCheck) to test diverse LLM-based false alarm reduction techniques, including hybrid approaches with static analysis. Key findings show that hybrid methods like LLM4SA and LLM4PFA can eliminate 94–98% of false positives with high recall, at modest time and cost, making them practical for industrial use. The study provides actionable guidance for practitioners and highlights research directions on knowledge-enhanced LLMs, long-context reasoning, and tighter integration with static analyzers. Limitations include challenges with long-context reasoning, complex cascaded constraints, and limited bug-type coverage, suggesting avenues for future work to broaden applicability across enterprise contexts.

Abstract

Static analysis tools (SATs) are widely adopted in both academia and industry for improving software quality, yet their practical use is often hindered by high false positive rates, especially in large-scale enterprise systems. These false alarms demand substantial manual inspection, creating severe inefficiencies in industrial code review. While recent work has demonstrated the potential of large language models (LLMs) for false alarm reduction on open-source benchmarks, their effectiveness in real-world enterprise settings remains unclear. To bridge this gap, we conduct the first comprehensive empirical study of diverse LLM-based false alarm reduction techniques in an industrial context at Tencent, one of the largest IT companies in China. Using data from Tencent's enterprise-customized SAT on its large-scale Advertising and Marketing Services software, we construct a dataset of 433 alarms (328 false positives, 105 true positives) covering three common bug types. Through interviewing developers and analyzing the data, our results highlight the prevalence of false positives, which wastes substantial manual effort (e.g., 10-20 minutes of manual inspection per alarm). Meanwhile, our results show the huge potential of LLMs for reducing false alarms in industrial settings (e.g., hybrid techniques of LLM and static analysis eliminate 94-98% of false positives with high recall). Furthermore, LLM-based techniques are cost-effective, with per-alarm costs as low as 2.1-109.5 seconds and $0.0011-$0.12, representing orders-of-magnitude savings compared to manual review. Finally, our case analysis further identifies key limitations of LLM-based false alarm reduction in industrial settings.

Figures (3)

• Figure 1: Overview of Data Collection
• Figure 2: Overview of Code Review in Tencent
• Figure 3: Comparison of performance for LLM-based methods