TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks

TL;DR

TimeMachine-bench introduces a scalable, live benchmark for repository-level software migration in real-world Python projects by leveraging date-based environment control to reproduce old and future dependency states. It automates the construction of a large Full dataset and a carefully curated Verified subset, enabling continuous evaluation across 11 models including strong open-weight and proprietary LLMs. The study reveals that while modern models show promise on migration tasks, reliability remains challenged by spurious edits, test-coverage gaps, and suboptimal tool use, prompting the need for improved evaluation protocols and richer verification. The framework’s live-updating design and granular metrics offer a practical path to advancing robust, evolution-aware code-migration capabilities in AI-assisted software engineering.

Abstract

With the advancement of automated software engineering, research focus is increasingly shifting toward practical tasks reflecting the day-to-day work of software engineers. Among these tasks, software migration, a critical process of adapting code to evolving environments, has been largely overlooked. In this study, we introduce TimeMachine-bench, a benchmark designed to evaluate software migration in real-world Python projects. Our benchmark consists of GitHub repositories whose tests begin to fail in response to dependency updates. The construction process is fully automated, enabling live updates of the benchmark. Furthermore, we curated a human-verified subset to ensure problem solvability. We evaluated agent-based baselines built on top of 11 models, including both strong open-weight and state-of-the-art LLMs on this verified subset. Our results indicated that, while LLMs show some promise for migration tasks, they continue to face substantial reliability challenges, including spurious solutions that exploit low test coverage and unnecessary edits stemming from suboptimal tool-use strategies. Our dataset and implementation are available at https://github.com/tohoku-nlp/timemachine-bench.

Figures (13)

• Figure 1: Overview of TimeMachine-bench. Using date-based environment control, our framework enables strict reproduction of two distinct environments corresponding to specific points in time.
• Figure 2: Construction pipeline of TimeMachine-bench. Steps 1--4 are fully automated to ensure the benchmark's scalability and live nature, resulting in the Full dataset of 1,145 repositories. Step 5 incorporates human verification to curate the high-quality Verified subset of 100 repositories.
• Figure 3: Example task from TimeMachine-bench-Verified. In this case, the model is asked to handle multiple runtime errors in stages, triggered by an update to the pandas library (Difficulty = Easy).
• Figure 4: List of libraries that triggered the errors in the Verified subset. The total does not sum up to 100 as some repositories had issues spanning multiple libraries.
• Figure 5: $\text{pass@1}(n', 10)$ on TimeMachine-bench-Verified dataset with varying $n'$ values.