Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

JMigBench: A Benchmark for Evaluating LLMs on Source Code Migration (Java 8 to Java 11)

Nishil Amin, Zhiwei Fei, Xiang Li, Justyna Petke, He Ye

TL;DR

This work presents JMigBench, a function-level benchmark for evaluating LLM-driven migrations from Java 8 to Java 11, addressing a gap where existing benchmarks focus on repository-level changes. The study constructs a ground-truth dataset across eight deprecated API categories, initially encountering noise and bias that were mitigated by curating a revised 45-function set. Using CodeBLEU and a keyword-removal metric, the Mistral Codestral model showed moderate lexical and semantic alignment with ground-truth Java 11 code and perfect migrations in only 11.11% of cases, with limited removal of certain deprecated APIs (notably CORBA and JAX-WS). The benchmark and findings reveal that LLMs can reduce developer effort for simple migrations but cannot yet replace human expertise for complex transformations, highlighting avenues for dataset expansion, prompting strategies, and cross-model evaluation in future work.

Abstract

We build a benchmark to evaluate large language models (LLMs) for source code migration tasks, specifically upgrading functions from Java 8 to Java 11. We first collected a dataset of function pairs from open-source repositories, but limitations in data quality led us to construct a refined dataset covering eight categories of deprecated APIs. Using this dataset, the Mistral Codestral model was evaluated with CodeBLEU and keyword-based metrics to measure lexical and semantic similarity as well as migration correctness. Results show that the evaluated model (Mistral Codestral) can handle trivial one-to-one API substitutions with moderate success, achieving identical migrations in 11.11% of the cases, but it struggles with more complex migrations such as CORBA or JAX-WS. These findings suggest Mistral Codestral can partially reduce developer effort by automating repetitive migration tasks but cannot yet replace humans within the scope of the JMigBench benchmark. The benchmark and analysis provide a foundation for future work on expanding datasets, refining prompting strategies, and improving migration performance across different LLMs.

JMigBench: A Benchmark for Evaluating LLMs on Source Code Migration (Java 8 to Java 11)

TL;DR

This work presents JMigBench, a function-level benchmark for evaluating LLM-driven migrations from Java 8 to Java 11, addressing a gap where existing benchmarks focus on repository-level changes. The study constructs a ground-truth dataset across eight deprecated API categories, initially encountering noise and bias that were mitigated by curating a revised 45-function set. Using CodeBLEU and a keyword-removal metric, the Mistral Codestral model showed moderate lexical and semantic alignment with ground-truth Java 11 code and perfect migrations in only 11.11% of cases, with limited removal of certain deprecated APIs (notably CORBA and JAX-WS). The benchmark and findings reveal that LLMs can reduce developer effort for simple migrations but cannot yet replace human expertise for complex transformations, highlighting avenues for dataset expansion, prompting strategies, and cross-model evaluation in future work.

Abstract

We build a benchmark to evaluate large language models (LLMs) for source code migration tasks, specifically upgrading functions from Java 8 to Java 11. We first collected a dataset of function pairs from open-source repositories, but limitations in data quality led us to construct a refined dataset covering eight categories of deprecated APIs. Using this dataset, the Mistral Codestral model was evaluated with CodeBLEU and keyword-based metrics to measure lexical and semantic similarity as well as migration correctness. Results show that the evaluated model (Mistral Codestral) can handle trivial one-to-one API substitutions with moderate success, achieving identical migrations in 11.11% of the cases, but it struggles with more complex migrations such as CORBA or JAX-WS. These findings suggest Mistral Codestral can partially reduce developer effort by automating repetitive migration tasks but cannot yet replace humans within the scope of the JMigBench benchmark. The benchmark and analysis provide a foundation for future work on expanding datasets, refining prompting strategies, and improving migration performance across different LLMs.
Paper Structure (22 sections, 1 equation, 4 figures, 2 tables)

This paper contains 22 sections, 1 equation, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Code Migration Benchmarks
  4. Existing Migration Techniques
  5. JMig Bench: Benchmark for Java 8 to 11 Migration
  6. Dataset Collection
  7. Quality Checking and Synthesizing
  8. Experimental Methodology
  9. Research Questions
  10. Methodology of RQ1
  11. Methodology of RQ2
  12. Experimental Settings
  13. Experimental Results
  14. RQ1: Lexical and Semantic Similarity
  15. CodeBLEU Metrics
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Function Length Boxplot (Final Dataset)
  • Figure 2: Distribution of Deprecated Terms (Final Dataset)
  • Figure 3: Prompt messages used for all experiments
  • Figure 4: Bar Chart showing the keyword removal success