Utilizing Source Code Syntax Patterns to Detect Bug Inducing Commits using Machine Learning Models

TL;DR

The paper addresses Bug Inducing Commit detection by introducing syntax-pattern-based features (Token Sequence and Token Pattern) derived from commit patch source code using srcML, and evaluates their effectiveness against conventional GitHub-statistics features across eight open-source projects in Java, C++, and Python. It systematically encodes these patterns as ML-ready features, selects the most informative ones per project, and benchmarks five classifiers with a time-aware split, showing consistent improvements in Precision, Recall, F1, and AUC. Moreover, the study demonstrates improved explainability for TP-based features via PyExplainer, suggesting these patterns offer clearer insights into why a commit may be buggy. The findings indicate that developers' coding syntax patterns carry predictive signal for bug proneness and highlight potential for integrating syntax-pattern analysis into JIT defect prediction tools and IDE-assisted bug-prevention workflows.

Abstract

Detecting Bug Inducing Commit (BIC) or Just in Time (JIT) defect prediction using Machine Learning (ML) based models requires tabulated feature values extracted from the source code or historical maintenance data of a software system. Existing studies have utilized meta-data from source code repositories (we named them GitHub Statistics or GS), n-gram-based source code text processing, and developer's information (e.g., the experience of a developer) as the feature values in ML-based bug detection models. However, these feature values do not represent the source code syntax styles or patterns that a developer might prefer over available valid alternatives provided by programming languages. This investigation proposed a method to extract features from its source code syntax patterns to represent software commits and investigate whether they are helpful in detecting bug proneness in software systems. We utilize six manually and two automatically labeled datasets from eight open-source software projects written in Java, C++, and Python programming languages. Our datasets contain 642 manually labeled and 4,014 automatically labeled buggy and non-buggy commits from six and two subject systems, respectively. The subject systems contain a diverse number of revisions, and they are from various application domains. Our investigation shows the inclusion of the proposed features increases the performance of detecting buggy and non-buggy software commits using five different machine learning classification models. Our proposed features also perform better in detecting buggy commits using the Deep Belief Network generated features and classification model. This investigation also implemented a state-of-the-art tool to compare the explainability of predicted buggy commits using our proposed and traditional features and found that our proposed features provide better reasoning about buggy.....

Figures (13)

• Figure 1: Overall Steps of this Study
• Figure 2: Demonstration of Feature Extraction from the Source Code of Commit Patch
• Figure 3: Recursive Feature Elimination
• Figure 4: We are comparing the improvements of F1 Scores using different Feature Combinations using five different machine learning-based classifiers to predict buggy and non-buggy commits from two popular datasets (OpenStack and QT). Here, the horizontal line at 0 indicates the baseline F1 Scores, and the lines and dots above and below the horizontal line indicate the percentage of improvements and decreases in F1 Scores, respectively, when we use different combinations of our proposed features.
• Figure 5: Performance Comparison by Precision in different subject systems using different feature combinations.