Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Refactoring Commits in Machine Learning Python Projects: A Machine Learning-Based Approach

Shayan Noei, Heng Li, Ying Zou

TL;DR

This paper introduces MLRefScanner, a learning-based approach to detect refactoring commits in ML Python projects, addressing the gap left by AST-pattern tools in Python and ML contexts. By combining textual, process, and code features with undersampling and a diverse set of classifiers, the method achieves a high precision of 0.94 and recall of 0.82 in mixed-project evaluations, with AUC up to 0.89, and improves further when ensembled with the PyRef rule-based tool (precision 0.95, recall 0.99). The authors provide a carefully labeled dataset from 199 ML repositories, analyze feature importance via LIME, and demonstrate that textual signals dominate while process and code features provide complementary gains. They also show that the approach generalizes to general Python projects and can augment existing refactoring detectors through ensemble strategies, offering practical benefits for maintainability analytics and tool development in ML software ecosystems.

Abstract

Refactoring enhances software quality without altering its functional behaviors. Understanding the refactoring activities of developers is crucial to improving software maintainability. With the increasing use of machine learning (ML) libraries and frameworks, maximizing their maintainability is crucial. Due to the data-driven nature of ML projects, they often undergo different refactoring operations (e.g., data manipulation), for which existing refactoring tools lack ML-specific detection capabilities. Furthermore, a large number of ML libraries are written in Python, which has limited tools for refactoring detection. PyRef, a rule-based and state-of-the-art tool for Python refactoring detection, can identify 11 types of refactoring operations. In comparison, Rminer can detect 99 types of refactoring for Java projects. We introduce MLRefScanner, a prototype tool that applies machine-learning techniques to detect refactoring commits in ML Python projects. MLRefScanner identifies commits with both ML-specific and general refactoring operations. Evaluating MLRefScanner on 199 ML projects demonstrates its superior performance compared to state-of-the-art approaches, achieving an overall 94% precision and 82% recall. Combining it with PyRef further boosts performance to 95% precision and 99% recall. Our study highlights the potential of ML-driven approaches in detecting refactoring across diverse programming languages and technical domains, addressing the limitations of rule-based detection methods.

Detecting Refactoring Commits in Machine Learning Python Projects: A Machine Learning-Based Approach

TL;DR

This paper introduces MLRefScanner, a learning-based approach to detect refactoring commits in ML Python projects, addressing the gap left by AST-pattern tools in Python and ML contexts. By combining textual, process, and code features with undersampling and a diverse set of classifiers, the method achieves a high precision of 0.94 and recall of 0.82 in mixed-project evaluations, with AUC up to 0.89, and improves further when ensembled with the PyRef rule-based tool (precision 0.95, recall 0.99). The authors provide a carefully labeled dataset from 199 ML repositories, analyze feature importance via LIME, and demonstrate that textual signals dominate while process and code features provide complementary gains. They also show that the approach generalizes to general Python projects and can augment existing refactoring detectors through ensemble strategies, offering practical benefits for maintainability analytics and tool development in ML software ecosystems.

Abstract

Refactoring enhances software quality without altering its functional behaviors. Understanding the refactoring activities of developers is crucial to improving software maintainability. With the increasing use of machine learning (ML) libraries and frameworks, maximizing their maintainability is crucial. Due to the data-driven nature of ML projects, they often undergo different refactoring operations (e.g., data manipulation), for which existing refactoring tools lack ML-specific detection capabilities. Furthermore, a large number of ML libraries are written in Python, which has limited tools for refactoring detection. PyRef, a rule-based and state-of-the-art tool for Python refactoring detection, can identify 11 types of refactoring operations. In comparison, Rminer can detect 99 types of refactoring for Java projects. We introduce MLRefScanner, a prototype tool that applies machine-learning techniques to detect refactoring commits in ML Python projects. MLRefScanner identifies commits with both ML-specific and general refactoring operations. Evaluating MLRefScanner on 199 ML projects demonstrates its superior performance compared to state-of-the-art approaches, achieving an overall 94% precision and 82% recall. Combining it with PyRef further boosts performance to 95% precision and 99% recall. Our study highlights the potential of ML-driven approaches in detecting refactoring across diverse programming languages and technical domains, addressing the limitations of rule-based detection methods.
Paper Structure (36 sections, 2 equations, 3 figures, 7 tables)

This paper contains 36 sections, 2 equations, 3 figures, 7 tables.

Table of Contents

  1. introduction
  2. experiment setup
  3. Overview of Our Approach
  4. Subject Selection
  5. Labeling Refactoring Commits
  6. Keyword-based labeling
  7. Rule-based labeling
  8. Manual validation
  9. Feature Extraction
  10. Textual features
  11. Process features
  12. Code Features
  13. Feature pre-processing
  14. Model Construction
  15. Data splitting
  16. ...and 21 more sections

Figures (3)

  • Figure 1: Overview of the approach
  • Figure 2: Compare the Results of Different Undersampling Approaches
  • Figure 3: Compare the Performance of Our Approach Using Cross and within-projects Validations.