IRJIT: A Simple, Online, Information Retrieval Approach for Just-In-Time Software Defect Prediction

TL;DR

IRJIT proposes a simple, online information retrieval approach for Just-In-Time SDP that labels commits by BM25-based similarity to past buggy/clean changes and ranks buggy lines by tf-idf-inspired token importance, avoiding expensive retraining. It demonstrates competitive commit-level performance and superior explainability, linking predictions to historical commits and highlighting buggy lines without requiring hand-crafted features. Across ten open-source datasets, IRJIT achieves up to 112× speedups over ML/DL baselines and maintains practical accuracy, showing strong potential for real-time, resource-constrained development environments. The approach also emphasizes online adaptability, explainability at both commit and line levels, and a straightforward deployment path via commit hooks and Elasticsearch-based indexing, while acknowledging limitations related to SZZ reliance and the need for broader cross-project validation.

Abstract

Just-in-Time software defect prediction (JIT-SDP) prevents the introduction of defects into the software by identifying them at commit check-in time. Current software defect prediction approaches rely on manually crafted features such as change metrics and involve expensive to train machine learning or deep learning models. These models typically involve extensive training processes that may require significant computational resources and time. These characteristics can pose challenges when attempting to update the models in real-time as new examples become available, potentially impacting their suitability for fast online defect prediction. Furthermore, the reliance on a complex underlying model makes these approaches often less explainable, which means the developers cannot understand the reasons behind models' predictions. An approach that is not explainable might not be adopted in real-life development environments because of developers' lack of trust in its results. To address these limitations, we propose an approach called IRJIT that employs information retrieval on source code and labels new commits as buggy or clean based on their similarity to past buggy or clean commits. IRJIT approach is online and explainable as it can learn from new data without expensive retraining, and developers can see the documents that support a prediction, providing additional context. By evaluating 10 open-source datasets in a within project setting, we show that our approach is up to 112 times faster than the state-of-the-art ML and DL approaches, offers explainability at the commit and line level, and has comparable performance to the state-of-the-art.

Figures (11)

• Figure 1: Performance evolution of a typical JIT-SDP approach over time showing a declining trend. x-axis shows time step and y-axis shows G-mean values. The model was trained once and the subsequent predictions were made without retraining the model.
• Figure 2: An overview of IRJIT operation. IRJIT extracts past changes using SZZ and indexes them using inverted indexes. On arrival of a new commit, IRJIT classifies the commit based on its similarity with the past changes in the index. Finally, IRJIT ranks changed lines according to bugginess.
• Figure 3: Comparing CPU/GPU run time of the evaluated approaches in seconds. Only JITFine was evaluated on GPU. ORB is not included in the plot because of its negligible run time.
• Figure 4: Comparing G-mean and difference of recalls ($|$R0-R1$|$) performance of IRJIT$_{online}$ and ORB for all datasets. X-axis shows timesteps
• Figure 5: Density plot comparing the G-mean distribution of IRJIT$_{batched}$ with (a) JITLine$_{batched}$ and (b) JITFine$_{batched}$.