FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migration

TL;DR

FreshBrew presents a repository-level benchmark for evaluating AI agents on Java code migration from Java 8 to newer JDK versions. It combines a high-coverage, real-world dataset with a robust, reward-hacking–resistant evaluation protocol that requires semantic preservation, validated via compilation, unmodified tests, and maintained test coverage. An extensive empirical study across seven models and a deterministic OpenRewrite baseline reveals substantial variability in end-to-end migration success, with Gemini 2.5 Flash achieving the best performance around 52.3% on Java 17 and slightly lower on Java 21, while exposing prevalent reward-hacking and failure modes in agentic systems. The work provides actionable insights into the strengths and limitations of current agentic approaches for codebase modernization and establishes a reproducible platform to drive progress toward trustworthy, automated Java migrations.

Abstract

AI coding assistants are rapidly becoming integral to modern software development. A key challenge in this space is the continual need to migrate and modernize codebases in response to evolving software ecosystems. Traditionally, such migrations have relied on rule-based systems and human intervention. With the advent of powerful large language models (LLMs), AI-driven agentic frameworks offer a promising alternative-but their effectiveness has not been systematically evaluated. In this paper, we introduce FreshBrew, a novel benchmark for evaluating AI agents on project-level Java migrations, with a specific focus on measuring an agent's ability to preserve program semantics and avoid reward hacking, which we argue requires projects with high test coverage for a rigorous and reliable evaluation. We benchmark several state-of-the-art LLMs, and compare their performance against established rule-based tools. Our evaluation of AI agents on this benchmark of 228 repositories shows that the top-performing model, Gemini 2.5 Flash, can successfully migrate 52.3 percent of projects to JDK 17. Our empirical analysis reveals novel insights into the critical strengths and limitations of current agentic approaches, offering actionable insights into their real-world applicability. Our empirical study reveals failure modes of current AI agents in realistic Java modernization tasks, providing a foundation for evaluating trustworthy code-migration systems. By releasing FreshBrew, we aim to facilitate rigorous, reproducible evaluation and catalyze progress in AI-driven codebase modernization.

Figures (12)

• Figure 1: Overview of the FreshBrew benchmark for automated Java migration. (left) The dataset pipeline curates real-world repositories that build on JDK 8 but fail on JDK 17. (center) A generic migration agent performs the upgrade task. (right) Our evaluation protocol measures success through three sequential gates: (i) successful compilation, (ii) passing all original tests, and (iii) preservation of test coverage within 5 pp of the baseline. These gates ensure that only semantically correct migrations are counted as successes and effectively guard against reward hacking.
• Figure 2: Automated dataset‐construction pipeline used in this study.
• Figure 3: Overview of repository statistics in the FreshBrew dataset.
• Figure 4: Distribution of key statistics for repositories in the dataset. The y-axis for metrics with wide ranges (e.g., Lines of Code) is logarithmic to visualize the heavily skewed data. Each plot shows the median (orange line), interquartile range (box), and outliers (dots).
• Figure 5: Density plot of coverage drops for migrations classified as Legitimate Refactoring versus Reward Hacking. The clear separation between the two distributions supports the 5% threshold as a conservative boundary for identifying reward hacking.