Bridging Expert Knowledge with Deep Learning Techniques for Just-In-Time Defect Prediction

TL;DR

This work proposes SimCom++, a model fusion framework that adopts both early fusions on the feature level and late fusions on the decision level, and shows that SimCom++ can significantly outperform the baselines by 5.7–26.9%.

Abstract

Just-In-Time (JIT) defect prediction aims to automatically predict whether a commit is defective or not, and has been widely studied in recent years. In general, most studies can be classified into two categories: 1) simple models using traditional machine learning classifiers with hand-crafted features, and 2) complex models using deep learning techniques to automatically extract features from commit contents. Hand-crafted features used by simple models are based on expert knowledge but may not fully represent the semantic meaning of the commits. On the other hand, deep learning-based features used by complex models represent the semantic meaning of commits but may not reflect useful expert knowledge. Simple models and complex models seem complementary to each other to some extent. To utilize the advantages of both simple and complex models, we propose a model fusion framework that adopts both early fusions on the feature level and late fusions on the decision level. We propose SimCom++ by adopting the best early and late fusion strategies. The experimental results show that SimCom++ can significantly outperform the baselines by 5.7--26.9\%. In addition, our experimental results confirm that the simple model and complex model are complementary to each other.

Figures (12)

• Figure 1: The framework to combine simple and complex models. Given a commit, a simple model (e.g., Sim) deals with hand-made features based on expert knowledge, and a complex model (e.g., Com) extracts features from commit contents. The early fusion strategies can insert expert knowledge into a deep learning model to build another new expert knowledge-enhanced deep learning model. The late fusion strategies combine the prediction scores of the simple model, the complex model, and the model after early fusion, to give final predictions on the defectiveness of commits.
• Figure 2: Model Details of Com. Com separately encodes and extracts features from code changes and commit logs via different textCNN modules. For code changes, Com considers the hierarchical structure of code changes and extracts features from the line, hunk, file, and finally to the commit level. Com concatenates the commit log vector and code change vector to get the commit vector as the last representation used for prediction.
• Figure 3: Visualization of Early Fusion Techniques.
• Figure 4: Example of Late Fusion Strategies.
• Figure 5: Relations between the predictions of Sim and Com. The left Venn diagram shows the identified defective commits and the right one shows the false positives generated by Sim and Com.