RustEvo^2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation

TL;DR

RustEvo$^{\mathsf{2}}$ presents EvoEval, a two-phase framework to evaluate LLM-based Rust code generation under API evolution. By automatically collecting multi-source API changes and converting them into executable tasks, it yields a dataset of 588 changes across four categories, enabling rigorous, time-sensitive benchmarking. Experiments across ten SOTA models show clear performance gaps: stabilized APIs are easier to handle than behavioral changes, and models benefit from Retrieval-Augmented Generation, especially for APIs released after model knowledge cutoffs. The work demonstrates the necessity of evolution-aware benchmarks for maintaining LLM relevance in fast-changing software ecosystems and provides publicly available code and data to spur further research.

Abstract

Large Language Models (LLMs) have become pivotal tools for automating code generation in software development. However, these models face significant challenges in producing version-aware code for rapidly evolving languages like Rust, where frequent Application Programming Interfaces (API) changes across versions lead to compatibility issues and correctness errors. Existing benchmarks lack systematic evaluation of how models navigate API transitions, relying on labor-intensive manual curation and offering limited version-specific insights. To address this gap, we present RustEvo, a novel framework for constructing dynamic benchmarks that evaluate the ability of LLMs to adapt to evolving Rust APIs. RustEvo automates dataset creation by synthesizing 588 API changes (380 from Rust standard libraries, 208 from 15 third-party crates) into programming tasks mirroring real-world challenges. These tasks cover four API evolution categories: Stabilizations, Signature Changes, Behavioral Changes, and Deprecations, reflecting their actual distribution in the Rust ecosystem. Experiments on state-of-the-art (SOTA) LLMs reveal significant performance variations: models achieve a 65.8% average success rate on stabilized APIs but only 38.0% on behavioral changes, highlighting difficulties in detecting semantic shifts without signature alterations. Knowledge cutoff dates strongly influence performance, with models scoring 56.1% on before-cutoff APIs versus 32.5% on after-cutoff tasks. Retrieval-Augmented Generation (RAG) mitigates this gap, improving success rates by 13.5% on average for APIs released after model training. Our findings underscore the necessity of our evolution-aware benchmarks to advance the adaptability of LLMs in fast-paced software ecosystems. The framework and the benchmarks are publicly released at https://github.com/SYSUSELab/RustEvo.

Figures (7)

• Figure 1: Timeline illustrating the release dates of Rust versions and knowledge cutoff points of LLMs (2023–2025).
• Figure 2: A Motivating Example
• Figure 3: Two-phase RustEvo$^{\mathsf{2}}$ Construction Framework
• Figure 4: Query and signature generation examples corresponding to four API change categories: Stabilizations, Signature Changes, Behavioral Changes, and Deprecations.
• Figure 5: The EvoEval pipeline illustrated through an example.