MigrationBench: Repository-Level Code Migration Benchmark from Java 8

TL;DR

MigrationBench tackles the challenge of evaluating large-scale, repository-wide code migration with LLMs by introducing a Java-centric benchmark that migrates from Java 8 to Java 17/21. It provides a full dataset (5,102 repos), a UTG complement (4,814 repos), and a curated 300-repo selected subset, plus an automated evaluation framework and the SD-Feedback approach. Evaluation hinges on approximate functional equivalence using mvn clean verify, compiled class major-version checks, and test-method invariants, with defined minimal and maximal migration efficacies. Experimental results show Claude-3.5-Sonnet-v2 achieving the best minimal migration performance on the selected subset (62.33%), while maximal migrations remain more challenging (27.33%), underscoring the complexity of repository-wide upgrades and dependency versioning. Overall, MigrationBench provides a reproducible, scalable resource to study and advance LLM-assisted repository migrations, enabling systematic benchmarking and future improvements in migration strategies and test-generation methods.

Abstract

With the rapid advancement of powerful large language models (LLMs) in recent years, a wide range of software engineering tasks can now be addressed using LLMs, significantly enhancing productivity and scalability. Numerous benchmark datasets have been developed to evaluate the coding capabilities of these models, while they primarily focus on code generation and issue-resolution tasks. In contrast, we introduce a new coding benchmark MigrationBench with a distinct focus: code migration. MigrationBench aims to serve as a comprehensive benchmark for migration from Java $8$ to the latest long-term support (LTS) versions (Java $17$, $21$), including a full dataset and its subset selected with $5,102$ and $300$ repositories respectively. Selected is a representative subset curated for complexity and difficulty, offering a versatile resource to support research in the field of code migration. Additionally, we provide a comprehensive evaluation framework to facilitate rigorous and standardized assessment of LLMs on this challenging task. We further propose SD-Feedback and demonstrate that LLMs can effectively tackle repository-level code migration to Java $17$. For the selected subset with Claude-3.5-Sonnet-v2, SD-Feedback achieves $62.33\%$ and $27.33\%$ success rate (pass@1) for minimal and maximal migration respectively. The benchmark dataset and source code are available at: https://huggingface.co/collections/AmazonScience/migrationbench-68125452fc21a4564b92b6c3 and https://github.com/amazon-science/MigrationBench respectively.

Figures (8)

• Figure 1: The flowchart to find out the base commit id $H_b$ for code migration, starting with a Github repository $R$'s latest snapshot as of March $2024$. $^\dagger$ It assumes both the explicitly hard coded java versions and complied class major versions are monotonic. $^\ddagger$ Given the commit history of $V$ based on all pom.xml files, one can infer the first candidate snapshot $S_1$. $^\S$$H_b$ is the commit id of the last$S_1$ where java version $V$ passes version check, mvn command succeeds and compiled class major version $U$ passes version check.
• Figure 2: Minimal and maximal migration efficacy comparison (with either mvn clean verify or compile), as a function of number of SD-Feedback iterations, for the migration-bench-selected dataset using Claude-3.5-Sonnet-v2.
• Figure 3: Discrete probability density function comparison for number of modules, LOC, all or test Java files and test cases for repositories in MigrationBench's full and selected subsets.
• Figure 4: Cumulative distribution function comparison for number of modules, LOC, all or test-only Java files and test cases (dropping invalid negative values) for repositories in MigrationBench's full and selected subsets.
• Figure 5: SD-Feedback flowchart to resolve code migration errors, potentially with chain of feedback to LLMs. $^\dag$ The reported success might be inconsistent with that in the final evaluation, which may include extra checks as shown in \ref{['sec:eval']}.