LLM4SZZ: Enhancing SZZ Algorithm with Context-Enhanced Assessment on Large Language Models
Lingxiao Tang, Jiakun Liu, Zhongxin Liu, Xiaohu Yang, Lingfeng Bao
TL;DR
The paper addresses the challenge of accurately identifying bug-inducing commits in the SZZ framework by introducing LLM4SZZ, which adaptively combines context-enhanced assessment and two identification strategies based on the LLM’s understanding of the bug. It leverages bug-fixing commits and commit messages, uses context expansion and refined prompts (with hints) to guide the LLM, and selects between context-enhanced or rank-based identification per case. Across three high-quality datasets, LLM4SZZ consistently outperforms traditional baselines in $F1$-score with improved precision and robust recall, and its core ideas transfer to open-source LLMs like llama3-8b and llama3-70b. The work highlights both the promise and challenges of integrating LLMs into SZZ workflows, including noise handling, scalability, and the handling of large or subtle bugs, and it points to future extensions across languages and fine-tuning opportunities for even stronger performance.
Abstract
The SZZ algorithm is the dominant technique for identifying bug-inducing commits and serves as a foundation for many software engineering studies, such as bug prediction and static code analysis. Researchers have proposed many variants to enhance the SZZ algorithm's performance since its introduction. The majority of them rely on static techniques or heuristic assumptions, making them easy to implement, but their performance improvements are often limited. Recently, a deep learning-based SZZ algorithm has been introduced to enhance the original SZZ algorithm. However, it requires complex preprocessing and is restricted to a single programming language. Additionally, while it enhances precision, it sacrifices recall. Furthermore, most of variants overlook crucial information, such as commit messages and patch context, and are limited to bug-fixing commits involving deleted lines. The emergence of large language models (LLMs) offers an opportunity to address these drawbacks. In this study, we investigate the strengths and limitations of LLMs and propose LLM4SZZ, which employs two approaches (i.e., rank-based identification and context-enhanced identification) to handle different types of bug-fixing commits. We determine which approach to adopt based on the LLM's ability to comprehend the bug and identify whether the bug is present in a commit. The context-enhanced identification provides the LLM with more context and requires it to find the bug-inducing commit among a set of candidate commits. In rank-based identification, we ask the LLM to select buggy statements from the bug-fixing commit and rank them based on their relevance to the root cause. Experimental results show that LLM4SZZ outperforms all baselines across three datasets, improving F1-score by 6.9% to 16.0% without significantly sacrificing recall.