The Repeat Offenders: Characterizing and Predicting Extremely Bug-Prone Source Methods

TL;DR

The paper investigates ExtremelyBuggy methods—source methods repeatedly involved in bug fixes—across 1.25 million methods from 98 open-source Java projects. It combines large-scale method-history data (via CodeShovel), bug-proneness labeling, and 14 code metrics to assess prevalence, quality, and early predictability, complemented by a thematic qualitative analysis of 265 ExtremelyBuggy methods. The findings show ExtremelyBuggy methods are a small minority yet account for a large share of bugs, and while they exhibit detectable quality deficiencies at inception, simple metric-based predictions are unreliable due to class imbalance and cross-project variability. The work provides a public dataset, actionable patterns of visual, contextual, and defect themes, and practical guidance for researchers and practitioners to target high-risk methods with evolution-aware and hybrid modeling approaches.

Abstract

Identifying the small subset of source code that repeatedly attracts bugs is critical for reducing long-term maintenance effort. We define ExtremelyBuggy methods as those involved in more than one bug fix and present the first large-scale study of their prevalence, characteristics, and predictability. Using a dataset of over 1.25 million methods from 98 open-source Java projects, we find that ExtremelyBuggy methods constitute only a tiny fraction of all methods, yet frequently account for a disproportionately large share of bugs. At their inception, these methods are significantly larger, more complex, less readable, and less maintainable than both singly-buggy and non-buggy methods. However, despite these measurable differences, a comprehensive evaluation of five machine learning models shows that early prediction of ExtremelyBuggy methods remains highly unreliable due to data imbalance, project heterogeneity, and the fact that many bugs emerge through subsequent evolution rather than initial implementation. To complement these quantitative findings, we conduct a thematic analysis of 265 ExtremelyBuggy methods, revealing recurring visual issues (e.g., confusing control flow, poor readability), contextual roles (e.g., core logic, data transformation, external resource handling), and common defect patterns (e.g., faulty conditionals, fragile error handling, misuse of variables). These results highlight the need for richer, evolution-aware representations of code and provide actionable insights for practitioners seeking to prioritize high-risk methods early in the development lifecycle.

Figures (6)

• Figure 1: Both figures use the HighRecall Dataset to measure bugs. To enhance readability, we only use 11 markers when presenting each distribution, however, all data points are represented.
• Figure 2: CDFs of the percent of methods in that are Buggy and ExtremelyBuggy for each of the 3 datasets for all projects. To enhance readability, we only use 11 markers when presenting each distribution, however, all 92 projects are represented.
• Figure 3: CDFs of the percent of bugs covered by the ExtremelyBuggy methods for each of the three datasets for all 92 projects, excluding projects with no ExtremelyBuggy methods. To enhance readability, we only use 11 markers when presenting each distribution.
• Figure 4: CDFs showing the distributions of different code metrics at the first commit between NotBuggy, Buggy, and ExtremelyBuggy methods for the Balanced dataset. To enhance readability, we only use 11 markers when presenting each distribution; however, all methods are represented. Note that 34 of the 700000 methods had a MaintainabilityIndex below zero. To improve readability, we excluded these methods when constructing the MaintainabilityIndex graph.
• Figure 5: Results for two of the select machine learning algorithms for the High Recall dataset. Results are very similar with other algorithms, and the results from the other algorithms can be viewed in our publicly shared repository . To enhance readability, we use less number of markers than the original number of projects.