A Novel Refactoring and Semantic Aware Abstract Syntax Tree Differencing Tool and a Benchmark for Evaluating the Accuracy of Diff Tools
Pouria Alikhanifard, Nikolaos Tsantalis
TL;DR
This work addresses the difficulty of understanding code evolution by introducing a novel AST diff tool built on RefactoringMiner that is refactoring-aware, language-aware, and capable of commit-level diffs. It extends RefactoringMiner 2.0 to 3.0 with improved statement mappings, multi-mapping support, and semantic imports, and couples this with a publicly available ground-truth benchmark (DiffBenchmark) comprising 800 bug-fixing and 188 refactoring commits. Extensive evaluation against GumTree variants, IJM, MTDiff, and srcDiff shows that the proposed tool achieves substantially higher precision and recall, especially on refactoring commits, and maintains competitive execution times. The benchmark and methodology enable objective, multi-dimensional assessment of AST diff tools, highlighting the advantages of language-aware and refactoring-aware approaches for accurate code-diff generation and analysis.
Abstract
Software undergoes constant changes to support new requirements, address bugs, enhance performance, and ensure maintainability. Thus, developers spend a great portion of their workday trying to understand and review the code changes of their teammates. Abstract Syntax Tree (AST) diff tools were developed to overcome the limitations of line-based diff tools, which are used by the majority of developers. Despite the notable improvements brought by AST diff tools in understanding complex changes, they still suffer from serious limitations, such as (1) lacking multi-mapping support, (2) matching semantically incompatible AST nodes, (3) ignoring language clues to guide the matching process, (4) lacking refactoring awareness, and (5) lacking commit-level diff support. We propose a novel AST diff tool based on RefactoringMiner that resolves all aforementioned limitations. First, we improved RefactoringMiner to increase its statement mapping accuracy, and then we developed an algorithm that generates AST diff for a given commit or pull request based on the refactoring instances and pairs of matched program element declarations provided by RefactoringMiner. To evaluate the accuracy of our tool and compare it with the state-of-the-art tools, we created the first benchmark of AST node mappings, including 800 bug-fixing commits and 188 refactoring commits. Our evaluation showed that our tool achieved a considerably higher precision and recall, especially for refactoring commits, with an execution time that is comparable with that of the faster tools.