Contrasting the Hyperparameter Tuning Impact Across Software Defect Prediction Scenarios

TL;DR

This paper directly contrasts the impact of hyperparameter tuning on software defect prediction across two evaluation scenarios: Inner Version Defect Prediction (IVDP) and Cross Version Defect Prediction (CVDP). Using a large-scale setup with 28 ML algorithms, 53 post-release datasets, grid and random search tuning, and five performance metrics, the authors assess how tuning gains vary with the SDP scenario, dataset size, and algorithm. Key findings show larger tuning benefits in IVDP than in CVDP, notable instances of negative tuning effects in CVDP, and greater tuning variability on small datasets. The study offers scenario-aware guidelines for practitioners and highlights the need to consider multiple SDP scenarios when evaluating tuning methods to ensure practical relevance and cost-effective deployment.

Abstract

Software defect prediction (SDP) is crucial for delivering high-quality software products. Recent research has indicated that prediction performance improvements in SDP are achievable by applying hyperparameter tuning to a particular SDP scenario. However, the positive impact resulting from the hyperparameter tuning step may differ based on the targeted SDP scenario. Comparing the impact of hyperparameter tuning across SDP scenarios is necessary to provide comprehensive insights and enhance the robustness, generalizability, and, eventually, the practicality of SDP modeling for quality assurance. Therefore, in this study, we contrast the impact of hyperparameter tuning across two pivotal and consecutive SDP scenarios: (1) Inner Version Defect Prediction (IVDP) and (2) Cross Version Defect Prediction (CVDP). The main distinctions between the two scenarios lie in the scope of defect prediction and the selected evaluation setups. This study's experiments use common evaluation setups, 28 machine learning (ML) algorithms, 53 post-release software datasets, two tuning algorithms, and five optimization metrics. We apply statistical analytics to compare the SDP performance impact differences by investigating the overall impact, the single ML algorithm impact, and variations across different software dataset sizes. The results indicate that the SDP gains within the IVDP scenario are significantly larger than those within the CVDP scenario. The results reveal that asserting performance gains for up to 24 out of 28 ML algorithms may not hold across multiple SDP scenarios. Furthermore, we found that small software datasets are more susceptible to larger differences in performance impacts. Overall, the study findings recommend software engineering researchers and practitioners to consider the effect of the selected SDP scenario when expecting performance gains from hyperparameter tuning.

Figures (7)

• Figure 1: Overview of the included SDP scenarios experimental setups.
• Figure 2: Distributions of the values of $I(M_{\text{tuned}}, M_{\text{default}})$ --- defined in the Equation (\ref{['eq:perImpact1']}) --- with grid and random tunings for the IVDP and CVDP scenarios. The $x$-axis on the plots represents the distributions' density. The $M$ in this equation for the optimized metrics starts from the AUC Impact panel on the left and ends at the Accuracy Impact panel. The black lines represent the median values of the distributions. The * represents statistically significant results for the Mann-Whitney U test with $p < 0.05$; $(-)$ denotes $p \ge 0.05$. N represents negligible effect (based on the absolute value of Cliff's delta), S --- small effects, M --- medium effect, and L --- large effect. For clarity, outliers are removed from the boxplots.
• Figure 3: Percentage of negative impact with grid tuning for the IVDP and CVDP scenarios. The $y$-axis on the plot represents the percentage of cases where the tuning led to a negative impact on the performance across different optimization metrics.
• Figure 4: For the CVDP scenario and AUC as optimization metric, the top panels show the frequency of negative performance impacts (i.e., the number of times tuning led to a performance drop) for each ML algorithm (left) and each dataset (right). The bottom panels display the distribution of these negative impacts using boxplots. Each boxplot summarizes the magnitude and spread of negative impacts observed. This figure highlights the ML algorithms and datasets that are more prone to performance degradation when hyperparameter tuning is applied for the CVDP scenario.
• Figure 5: Distributions of the performance impact $I(AUC_{\text{tuned}}, AUC_{\text{default}})$ for each studied ML algorithm --- defined in the Equation (\ref{['eq:perImpact1']}) --- with grid tuning for the IVDP and CVDP Scenarios. The $x$-axis on the plots represents the distributions' density. The black lines represent the median values of the distributions. The * represents statistically significant results for the Mann-Whitney U test with $p < 0.05$; $(-)$ denotes $p \ge 0.05$. N represents negligible effect (based on the absolute value of Cliff's delta), S --- small effects, M --- medium effect, and L --- large effect. The red dashed line marks the zero impact value. The y-axis uses the Symlog scale. For clarity, outliers are removed from the boxplots.