CodeFlowLM: Incremental Just-In-Time Defect Prediction with Pretrained Language Models and Exploratory Insights into Defect Localization

TL;DR

CodeFlowLM demonstrates that incremental fine-tuning of pretrained code language models yields robust JIT-SDP performance under drift, latency, and imbalance, outperforming traditional online baselines in both within- and cross-project settings. The work extends to JIT-DL, where LLMs like GPT-5 show competitive recall and stability, though attention-based models still lead in fine-grained ranking. A thorough qualitative analysis reveals conservative bias, context limitations, and dataset labeling noise as primary causes of errors, informing prompt design and data curation. Overall, the paper provides strong empirical evidence for PLMs in continual JIT-SDP and offers actionable insights into the current limits of prompt-based defect reasoning for localization.

Abstract

This work introduces CodeFlowLM, an incremental learning framework for Just-In-Time Software Defect Prediction (JIT-SDP) that leverages pre-trained language models (PLMs). Unlike traditional online learners, CodeFlowLM employs continual fine-tuning to address concept drift, class imbalance, and verification latency without retraining from scratch. We evaluated encoder-only and encoder-decoder PLMs (notably CodeT5+ and UniXCoder) in JIT-SDP scenarios within and between projects, comparing them with the incremental baseline BORB. The results show that CodeFlowLM achieves up to 68% G-Mean gains, confirming its superior adaptability and robustness in evolving software environments. We further extend the analysis to Just-in-Time Defect Localization (JIT-DL), benchmarking Large Language Models (LLMs) such as GPT-5, Claude Sonnet 4.5, and Gemini 2.5 Pro against attention-based models. GPT-5 delivers comparable performance for Recall@20% and Effort@20% with higher stability, although attention-based methods retain an advantage in fine-grained ranking metrics (Top-k, IFA). A qualitative error analysis reveals that most false positives arise from (1) human-like conservative bias, (2) insufficient contextual information in diff-based prompts, and (3) potential dataset mislabeling in JIT-Defects4J. These findings highlight both the promise and the current limitations of LLM reasoning in defect localization. False negatives occur in smaller proportions. Overall, CodeFlowLM significantly advances the state of the art in incremental JIT-SDP, demonstrating superior adaptability and robustness in evolving software environments. Furthermore, our exploratory analysis of LLMs in JIT-DL not only benchmarks their performance against established attention-based models but also provides critical insights into the current limitations of prompt-based defect reasoning.