Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-Based Detection of Tangled Code Changes for Higher-Quality Method-Level Bug Datasets

Md Nahidul Islam Opu, Shaowei Wang, Shaiful Chowdhury

TL;DR

This work tackles noise from tangled changes in method-level bug datasets by leveraging Large Language Models (LLMs) to detect whether method-level changes are related to bug fixes, using both commit messages and code diffs. It systematically compares zero-shot, few-shot, chain-of-thought prompting, and embedding-based classifiers, finding that combining commit messages with diffs yields the strongest performance (F1 up to 0.883 for prompting and up to 0.906 with embeddings). A manually curated gold dataset and a Less-Noisy dataset demonstrate that LLM-based untangling improves separability between buggy and non-buggy methods, with potential to enhance future method-level bug prediction. The work also provides replication data and shows that open-source LLMs can approach proprietary models in effectiveness, offering practical paths for cleaner bug datasets and improved prediction accuracy in real-world software maintenance tasks.

Abstract

Tangled code changes, commits that conflate unrelated modifications such as bug fixes, refactorings, and enhancements, introduce significant noise into bug datasets and adversely affect the performance of bug prediction models. Addressing this issue at a fine-grained, method-level granularity remains unexplored. This is critical to address, as recent bug prediction models, driven by practitioner demand, are increasingly focusing on finer granularity rather than traditional class- or file-level predictions. This study investigates the utility of Large Language Models (LLMs) for detecting tangled code changes by leveraging both commit messages and method-level code diffs. We formulate the problem as a binary classification task and evaluate multiple prompting strategies, including zero-shot, few-shot, and chain-of-thought prompting, using state-of-the-art proprietary LLMs such as GPT-5 and Gemini-2.0-Flash, and open-source models such as GPT-OSS-120B and CodeBERT. Our results demonstrate that combining commit messages with code diffs significantly enhances model performance, with the combined few-shot and chain-of-thought prompting achieving an F1-score of 0.883. Additionally, we explore machine learning models trained on LLM-generated embeddings, where a multi-layer perceptron classifier achieves superior performance (F1-score: 0.906, MCC: 0.807). Applying our approach to 49 open-source projects improves the distributional separability of code metrics between buggy and non-buggy methods, demonstrating the promise of LLMs for method-level commit untangling and potentially contributing to improving the accuracy of future bug prediction models.

LLM-Based Detection of Tangled Code Changes for Higher-Quality Method-Level Bug Datasets

TL;DR

This work tackles noise from tangled changes in method-level bug datasets by leveraging Large Language Models (LLMs) to detect whether method-level changes are related to bug fixes, using both commit messages and code diffs. It systematically compares zero-shot, few-shot, chain-of-thought prompting, and embedding-based classifiers, finding that combining commit messages with diffs yields the strongest performance (F1 up to 0.883 for prompting and up to 0.906 with embeddings). A manually curated gold dataset and a Less-Noisy dataset demonstrate that LLM-based untangling improves separability between buggy and non-buggy methods, with potential to enhance future method-level bug prediction. The work also provides replication data and shows that open-source LLMs can approach proprietary models in effectiveness, offering practical paths for cleaner bug datasets and improved prediction accuracy in real-world software maintenance tasks.

Abstract

Tangled code changes, commits that conflate unrelated modifications such as bug fixes, refactorings, and enhancements, introduce significant noise into bug datasets and adversely affect the performance of bug prediction models. Addressing this issue at a fine-grained, method-level granularity remains unexplored. This is critical to address, as recent bug prediction models, driven by practitioner demand, are increasingly focusing on finer granularity rather than traditional class- or file-level predictions. This study investigates the utility of Large Language Models (LLMs) for detecting tangled code changes by leveraging both commit messages and method-level code diffs. We formulate the problem as a binary classification task and evaluate multiple prompting strategies, including zero-shot, few-shot, and chain-of-thought prompting, using state-of-the-art proprietary LLMs such as GPT-5 and Gemini-2.0-Flash, and open-source models such as GPT-OSS-120B and CodeBERT. Our results demonstrate that combining commit messages with code diffs significantly enhances model performance, with the combined few-shot and chain-of-thought prompting achieving an F1-score of 0.883. Additionally, we explore machine learning models trained on LLM-generated embeddings, where a multi-layer perceptron classifier achieves superior performance (F1-score: 0.906, MCC: 0.807). Applying our approach to 49 open-source projects improves the distributional separability of code metrics between buggy and non-buggy methods, demonstrating the promise of LLMs for method-level commit untangling and potentially contributing to improving the accuracy of future bug prediction models.
Paper Structure (18 sections, 4 figures, 6 tables)

This paper contains 18 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Tangled Changes
  4. LLMs for Code Analysis & Classification
  5. Methodology
  6. Dataset
  7. Prompt Preparation
  8. Selecting and Querying LLMs
  9. Evaluation Metrics
  10. Experimental Setup
  11. Approach, Analysis & Results
  12. RQ1: Performance of zero-shot LLMs using code diff and commit message
  13. RQ2: Performance of different prompting techniques
  14. Case analysis: an example of the effectiveness of chain-of-thought approach in a semantically ambiguous commit
  15. RQ3: Performance of embedding-based machine learning models
  16. ...and 3 more sections

Figures (4)

  • Figure 1: The methodology for RQ1 and RQ2 consists of three parts: (a) dataset, (b) prompt preparation, and (c) LLM Query. The prompt setup for few-shot + chain-of-thought prompting consists of four instruction parts: (i) persona, (ii) task description, (iii) behavioral guidance, and (iv) output formatting. For Diff-only detection of RQ1, references to the commit message are excluded from everywhere. Other experiments use the full instructions, with the output formatting modified to require a single-word answer in non-chain-of-thought setups. Few-shot examples are included only in few-shot experiments.
  • Figure 2: Commit message and two methods modified in the same commit: the message indicates a bug fix, but only the method on the right contains the bug-fix change, while the method on the left is a non-bug-related tangled change. This was used as few-shot examples in our prompts.
  • Figure 3: The methodology for RQ3 has two main parts: (i) embedding generations and (ii) building ML models.
  • Figure 4: Distribution of code metrics for Buggy and NotBuggy methods in the Titan project across the Noisy and Less-Noisy datasets. Metric distributions differ more significantly in the Less-Noisy dataset.