On the calibration of Just-in-time Defect Prediction
Xhulja Shahini, Jone Bartel, Klaus Pohl
TL;DR
This work addresses the reliability of JIT DP prediction probabilities by evaluating the calibration of three state-of-the-art techniques (DeepJIT, LAPredict, CodeBERT4JIT) on QT and OPENSTACK datasets. It compares two post-calibration methods (Platt scaling and Temperature scaling) and uses standard calibration metrics (ECE, MCE, Brier score) plus reliability diagrams to quantify miscalibration. The findings show pervasive miscalibration across all models (ECE ranging roughly from 2% to 35%), with post-calibration yielding significant improvements for some models (notably DeepJIT and CodeBERT4JIT) but limited or inconsistent benefits for others (LApredict). The study highlights the practical importance of calibration when using prediction probabilities for prioritization in QA and suggests directions for design-time calibration and broader datasets to improve reliability in JIT DP systems.
Abstract
Just in time defect prediction (JIT DP) leverages ML to identify defect-prone code commits, enabling quality assurance (QA) teams to allocate resources more efficiently by focusing on commits that are most likely to contain defects. Although JIT DP techniques have introduced improvements in terms of predictive accuracy, they are still susceptible to misclassification errors such as false positives and negatives. This can lead to wasted resources or undetected defects, a particularly critical concern when QA resources are limited. To mitigate these challenges and preserve the practical utility of JIT DP tools, it becomes essential to estimate the reliability of the predictions, i.e., computing confidence scores. Such scores can help practitioners determine the trustworthiness of predictions and thus prioritize them efficiently. A simple approach to computing confidence scores is to extract, alongside each prediction, the corresponding prediction probabilities and use them as indicators of confidence. However, for these probabilities to reliably serve as confidence scores, the predictive model must be well-calibrated. This means that the prediction probabilities must accurately represent the true likelihood of each prediction being correct. Miscalibration, common in modern ML models, distorts probability scores such that they do not align with the actual correctness probability. In this study, we evaluate the calibration of three JIT DP techniques to determine whether and to what extent they exhibit poor calibration. Furthermore, we assess whether post-calibration methods can improve the calibration of existing JIT defect prediction models. Our results reveal that all evaluated JIT DP models exhibit some level of miscalibration, with ECE ranging from 2-35%. Furthermore, post-calibration methods do not consistently improve the calibration.