LineBreaker: Finding Token-Inconsistency Bugs with Large Language Models

TL;DR

The paper addresses Token-Inconsistency Bugs (TIBs), semantic bugs arising from semantically incorrect yet syntactically valid tokens, which are hard to detect with traditional analysis. It first conducts a measurement study of LLM capabilities, finding GPT-4 promising but limited by precision, cost, and distraction, motivating a cascaded detection approach. The authors then design LineBreaker, a multi-stage pipeline combining lightweight local LLMs with a state-of-the-art model to filter, verify, and highlight TIBs, achieving practical precision and cost. Real-world evaluation on Python and C repositories yields 123 new TIBs with 41 merges, demonstrating LineBreaker’s effectiveness, scalability, and generality. The work provides a practical framework and dataset for scalable semantic bug detection in real-world codebases.

Abstract

Token-inconsistency bugs (TIBs) involve the misuse of syntactically valid yet incorrect code tokens, such as misused variables and erroneous function invocations, which can often lead to software bugs. Unlike simple syntactic bugs, TIBs occur at the semantic level and are subtle - sometimes they remain undetected for years. Traditional detection methods, such as static analysis and dynamic testing, often struggle with TIBs due to their versatile and context-dependent nature. However, advancements in large language models (LLMs) like GPT-4 present new opportunities for automating TIB detection by leveraging these models' semantic understanding capabilities. This paper reports the first systematic measurement of LLMs' capabilities in detecting TIBs, revealing that while GPT-4 shows promise, it exhibits limitations in precision and scalability. Specifically, its detection capability is undermined by the model's tendency to focus on the code snippets that do not contain TIBs; its scalability concern arises from GPT-4's high cost and the massive amount of code requiring inspection. To address these challenges, we introduce \name, a novel and cascaded TIB detection system. \name leverages smaller, code-specific, and highly efficient language models to filter out large numbers of code snippets unlikely to contain TIBs, thereby significantly enhancing the system's performance in terms of precision, recall, and scalability. We evaluated \name on 154 Python and C GitHub repositories, each with over 1,000 stars, uncovering 123 new flaws, 45\% of which could be exploited to disrupt program functionalities. Out of our 69 submitted fixes, 41 have already been confirmed or merged.

Figures (11)

• Figure 1: An Example of ChatGPT detecting an existing bug. The variable query in the red rectangular is buggy.
• Figure 2: TIB detection using code-specific LLM.
• Figure 3: Example: a two-round prompt template (1/2FTCa).
• Figure 4: Code-specific models' infilling performance.
• Figure 5: Pipeline overview of LineBreaker.