An Audit of Machine Learning Experiments on Software Defect Prediction

TL;DR

Software defect prediction research from 2019–2023 exhibits wide variation in experimental design and reporting, with limited reproducibility across studies. The authors conduct a rigorous systematic technical audit using a 27-item reproducibility instrument and nine study-issues checks on 101 papers, revealing 427 issues in total and only one paper free of issues. Reproducibility is moderate on average (median 0.52 on a 0–1 scale), higher for journal than conference papers, and correlated with open access and longer papers, while data/code sharing remains inconsistent and many links are broken. The findings underscore the need for standardized validation, comprehensive reporting, and open science practices to improve credibility and practical impact in SDP research.

Abstract

Background: Machine learning algorithms are widely used to predict defect prone software components. In this literature, computational experiments are the main means of evaluation, and the credibility of results depends on experimental design and reporting. Objective: This paper audits recent software defect prediction (SDP) studies by assessing their experimental design, analysis, and reporting practices against accepted norms from statistics, machine learning, and empirical software engineering. The aim is to characterise current practice and assess the reproducibility of published results. Method: We audited SDP studies indexed in SCOPUS between 2019 and 2023, focusing on design and analysis choices such as outcome measures, out of sample validation strategies, and the use of statistical inference. Nine study issues were evaluated. Reproducibility was assessed using the instrument proposed by González Barahona and Robles. Results: The search identified approximately 1,585 SDP experiments published during the period. From these, we randomly sampled 101 papers, including 61 journal and 40 conference publications, with almost 50 percent behind paywalls. We observed substantial variation in research practice. The number of datasets ranged from 1 to 365, learners or learner variants from 1 to 34, and performance measures from 1 to 9. About 45 percent of studies applied formal statistical inference. Across the sample, we identified 427 issues, with a median of four per paper, and only one paper without issues. Reproducibility ranged from near complete to severely limited. We also identified two cases of tortured phrases and possible paper mill activity. Conclusions: Experimental design and reporting practices vary widely, and almost half of the studies provide insufficient detail to support reproduction. The audit indicates substantial scope for improvement.

Figures (15)

• Figure 1: Overall audit process
• Figure 2: Distribution of paper length by page count
• Figure 3: Distribution of paper citation counts (unnormalised)
• Figure 4: Distribution of paper citation counts normalised by year
• Figure 5: Histogram and density plot of paper reproducibility scores