Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing

Xuefeng Li, Nir Ben-Israel, Yotam Raz, Belal Ahmed, Doron Serebro, Antoine Raux

TL;DR

This work introduces a benchmarking framework for repository-level code modernization built on an implementation-agnostic evaluation paradigm, and utilizes an implementation-agnostic test suite to verify functional equivalence between source and target implementations.

Abstract

The evolution of AI coding agents has shifted the frontier from simple snippet completion to autonomous repository-level engineering. However, evaluating these agents remains ill-posed in general code repository generation, where the lack of deterministic ground truth leads to ambiguous metrics. Code modernization via automated translation offers a more rigorous alternative by providing a fixed ground truth -- the source repository; yet existing benchmarks are limited to small-scale repositories and rely on language-specific unit tests visible to the agent, allowing test-driven overfitting. We address these limitations by introducing a benchmarking framework for repository-level code modernization built on an implementation-agnostic evaluation paradigm. This framework is instantiated through RepoMod-Bench: a benchmark of 21 real-world repositories with standardized interfaces, spanning 8 programming languages. The benchmark contains 1.6M lines of code (LOC) and 11,616 tests, with repository sizes ranging from 14 to 211K LOC. By targeting repositories with standardized interfaces, we utilize an implementation-agnostic test suite to verify functional equivalence between source and target implementations. This black-box approach ensures verification remains consistent across languages, and our environment hides all test suites from agents to prevent test-driven shortcuts. Evaluating four state-of-the-art agent configurations reveals a sharp scaling collapse: average pass rates drop from 91.3% on projects under 10K LOC to 15.3% on projects exceeding 50K LOC. These results demonstrate that autonomous modernization at scale remains a significant open challenge. Our benchmark and code are available at https://github.com/Modelcode-ai/mcode-benchmark.

RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing

TL;DR

This work introduces a benchmarking framework for repository-level code modernization built on an implementation-agnostic evaluation paradigm, and utilizes an implementation-agnostic test suite to verify functional equivalence between source and target implementations.

Abstract

The evolution of AI coding agents has shifted the frontier from simple snippet completion to autonomous repository-level engineering. However, evaluating these agents remains ill-posed in general code repository generation, where the lack of deterministic ground truth leads to ambiguous metrics. Code modernization via automated translation offers a more rigorous alternative by providing a fixed ground truth -- the source repository; yet existing benchmarks are limited to small-scale repositories and rely on language-specific unit tests visible to the agent, allowing test-driven overfitting. We address these limitations by introducing a benchmarking framework for repository-level code modernization built on an implementation-agnostic evaluation paradigm. This framework is instantiated through RepoMod-Bench: a benchmark of 21 real-world repositories with standardized interfaces, spanning 8 programming languages. The benchmark contains 1.6M lines of code (LOC) and 11,616 tests, with repository sizes ranging from 14 to 211K LOC. By targeting repositories with standardized interfaces, we utilize an implementation-agnostic test suite to verify functional equivalence between source and target implementations. This black-box approach ensures verification remains consistent across languages, and our environment hides all test suites from agents to prevent test-driven shortcuts. Evaluating four state-of-the-art agent configurations reveals a sharp scaling collapse: average pass rates drop from 91.3% on projects under 10K LOC to 15.3% on projects exceeding 50K LOC. These results demonstrate that autonomous modernization at scale remains a significant open challenge. Our benchmark and code are available at https://github.com/Modelcode-ai/mcode-benchmark.
Paper Structure (53 sections, 6 figures, 6 tables)

This paper contains 53 sections, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. The Benchmark
  3. Benchmarking Framework
  4. Repository Selection
  5. Standardized Interfaces
  6. Real-World Provenance
  7. Existing Test Coverage
  8. Diverse Language Pairs
  9. Construction Pipeline
  10. Stage 1: Repository Selection
  11. Stage 2: Test Suite Creation
  12. Stage 3: Configuration
  13. Stage 4: Validation
  14. Benchmark Suite
  15. Experiments
  16. ...and 38 more sections

Figures (6)

  • Figure 1: RepoMod-Bench task overview. An AI agent receives a complete source implementation and must produce a functionally equivalent translation in the target language. Evaluation uses a hidden test suite that interacts only through standardized interfaces (such as CLI or REST API).
  • Figure 2: Benchmark construction pipeline. Each benchmark undergoes four stages: repository selection based on interface and coverage criteria, test suite creation with implementation-agnostic filtering, configuration file generation, and validation against the source implementation.
  • Figure 3: Benchmark distribution. Translation pairs across 8 programming languages.
  • Figure 4: Example benchmark.yml for the jq-gojq CLI benchmark. This file serves as the single source of truth for build commands, runtime configuration, and test specifications.
  • Figure 5: Standardized prompt template (v1) for CLI benchmarks. Bracketed fields are populated from configuration files.
  • ...and 1 more figures