The Hitchhiker's Guide to Program Analysis: A Journey with Large Language Models

TL;DR

This work tackles the inherent precision–scalability tension in static analysis by coupling it with large language models through LLift, an automated framework that employs post-constraint guided path analysis, progressive prompting, task decomposition, and self-validation. Across nearly 1,000 potential Use-Before-Initialization bugs in the Linux kernel, LLift achieves about 50% precision and recalls all known bugs, additionally uncovering 13 previously unknown kernel bugs. The approach demonstrates that LLM-assisted bug discovery can operate effectively on large, real-world datasets, offering a viable path toward scalable, accurate analysis in complex codebases. The authors also discuss open-source deployment, cross-model applicability, and future integration with static analysis to further improve robustness and coverage.

Abstract

Static analysis is a widely used technique in software engineering for identifying and mitigating bugs. However, a significant hurdle lies in achieving a delicate balance between precision and scalability. Large Language Models (LLMs) offer a promising alternative, as recent advances demonstrate remarkable capabilities in comprehending, generating, and even debugging code. Yet, the logic of bugs can be complex and require sophisticated reasoning and a large analysis scope spanning multiple functions. Therefore, at this point, LLMs are better used in an assistive role to complement static analysis. In this paper, we take a deep dive into the open space of LLM-assisted static analysis, using use-before-initialization (UBI) bugs as a case study. To this end, we develop LLift, a fully automated framework that interfaces with both a static analysis tool and an LLM. By carefully designing the framework and the prompts, we are able to overcome a number of challenges, including bug-specific modeling, the large problem scope, the non-deterministic nature of LLMs, etc. Tested in a real-world scenario analyzing nearly a thousand potential UBI bugs produced by static analysis, LLift demonstrates a potent capability, showcasing a reasonable precision (50%) and appearing to have no missing bugs. It even identified 13 previously unknown UBI bugs in the Linux kernel. This research paves the way for new opportunities and methodologies in using LLMs for bug discovery in extensive, real-world datasets.

Figures (9)

• Figure 1: Code snippet of sscanf and its usecase
• Figure 2: The overview of LLift. Start with the discarded cases by UBITect and determine whether these potential bugs are true or false.
• Figure 3: A typical type of potential UBI bug. For each suspicious variable $X$, we expect it to 1) have an initializer function that probably initializes $X$ and 2) use $X$.
• Figure 4: Example run of LLift. For each potential bug, LLift ($\Phi_{1}$) identifies its initializer, ($\Phi_{2}$) extracts the post-constraints of the initializer, and ($\Phi_{3}$) analyzes the behavior of the initializer with the post-constraints via LLM.
• Figure 5: The workflow of LLift. Given a potential bug, we let LLM first identify the initializer and then extract its post-constraints (Convo.1), then leverage them to summarize the behavior of the initializer (Convo.2). A conversation consists of prompts (boxes) and responses (edges).