One Bug, Hundreds Behind: LLMs for Large-Scale Bug Discovery

TL;DR

BugStone introduces a patch-centered, LLM-guided workflow to automatically identify Recurring Pattern Bugs (RPBs) across large codebases. It derives security coding rules from a seed patch, expands patch context with lightweight static analysis, and scans the codebase with LLM verification to surface related bugs. On a ground-truth dataset of $1{,}900$ patches and $80$ RPBs, it achieves $P=92.2\%$ and $PA=79.1\%$, and at scale with 135 rules it flags $22{,}568$ potential Linux kernel issues, of which $246$ are confirmed as true in a sample. The results demonstrate that a rule-centric, patch-seeded LLM approach can scale bug discovery and provide actionable findings for maintenance and security in large software systems.

Abstract

Fixing bugs in large programs is a challenging task that demands substantial time and effort. Once a bug is found, it is reported to the project maintainers, who work with the reporter to fix it and eventually close the issue. However, across the program, there are often similar code segments, which may also contain the bug, but were missed during discovery. Finding and fixing each recurring bug instance individually is labor intensive. Even more concerning, bug reports can inadvertently widen the attack surface as they provide attackers with an exploitable pattern that may be unresolved in other parts of the program. In this paper, we explore these Recurring Pattern Bugs (RPBs) that appear repeatedly across various code segments of a program or even in different programs, stemming from a same root cause, but are unresolved. Our investigation reveals that RPBs are widespread and can significantly compromise the security of software programs. This paper introduces BugStone, a program analysis system empowered by LLVM and a Large Language Model (LLM). The key observation is that many RPBs have one patched instance, which can be leveraged to identify a consistent error pattern, such as a specific API misuse. By examining the entire program for this pattern, it is possible to identify similar sections of code that may be vulnerable. Starting with 135 unique RPBs, BugStone identified more than 22K new potential issues in the Linux kernel. Manual analysis of 400 of these findings confirmed that 246 were valid. We also created a dataset from over 1.9K security bugs reported by 23 recent top-tier conference works. We manually annotate the dataset, identify 80 recurring patterns and 850 corresponding fixes. Even with a cost-efficient model choice, BugStone achieved 92.2% precision and 79.1% pairwise accuracy on the dataset.

Figures (3)

• Figure 1: The [0.5]create_singlethread_workqueue macro can return [0.5]NULL on failure. This patch adds a simple check to [0.5]nes_cm.c. It is invoked more than 197 times in other parts of the Linux kernel, but not every invocation checks the returned pointer value. This creates an RPB.
• Figure 2: Overview of BugStone.
• Figure 3: Average cost per detection (top) and unit price of LLMs (bottom) when using the Rule configuration prompt.