GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models

TL;DR

A novel, manually curated dataset comprising 116 Python code completion problems, each conditioned on specific library versions and accompanied by executable unit tests, designed to rigorously assess the ability of modern large language models (LLMs) to generate version-specific code that is not only syntactically correct but also functionally accurate upon execution.

Abstract

The rapid evolution of software libraries presents a significant challenge for code generation models, which must adapt to frequent version updates while maintaining compatibility with previous versions. Existing code completion benchmarks often overlook this dynamic aspect, and the one that does consider it relies on static code prediction tasks without execution-based evaluation, offering a limited perspective on a model's practical usability. To address this gap, we introduce \textbf{\GitChameleon{}}, a novel, manually curated dataset comprising 116 Python code completion problems, each conditioned on specific library versions and accompanied by executable unit tests. \GitChameleon{} is designed to rigorously assess the ability of modern large language models (LLMs) to generate version-specific code that is not only syntactically correct but also functionally accurate upon execution. Our comprehensive evaluations reveal that state-of-the-art LLMs struggle with this task; for instance, \textbf{GPT-4o} achieves a pass@10 of only 39.9\% (43.7\% when provided with error feedback), highlighting the complexity of the problem and the limitations of current models. By providing an execution-based benchmark that emphasizes the dynamic nature of code libraries, \GitChameleon{} serves as a critical tool to advance the development of more adaptable and reliable code generation models. For facilitation for further exploration of version-conditioned code generation, we make our code repository publicly accessible at \url{https://github.com/NizarIslah/GitChameleon}.

Figures (7)

• Figure 1: Left: Modern LLMs often struggle with generating version-accurate code, highlighting the need for benchmarks that specifically assess their ability to handle versioning. Right: Cumulative year-over-year (YoY) version releases of popular Python-based machine learning libraries show a consistent upward trend, reflecting the rapid pace of development and version updates of code libraries and packages.
• Figure 2: Fine-grained statistics of the GitChameleon benchmark. (a) The library with the most unique versions in the dataset is networkx with 8, whereas only 1 version of spacy and PyCaret are represented in GitChameleon. (b) Most versions in the dataset were released between 2021-2023, with a few versions released in earlier years. (c) The most common type of changes between versions were function name changes and argument/attribute changes, while semantic/output changes were least common.
• Figure 3: Correlation of GitChameleon with the representative code benchmarks (HumanEval, EvalPlus, and BigCodeBench-Hard split). For each benchmark, the spearman correlation coefficient was 0.37, 0.50, and 0.36, respectively.
• Figure 4: Instruct-tuned model performance breakdown by version release year (top) and type of change (bottom): We analyze model performance in terms of pass @ 10 for baseline and with error feedback generation across two dimensions: version release year and type of changes. Darker shaded bars represent values obtained via error feedback generation. Standard deviation is drawn as a black line, obtained from 5 random seeds. (Top) Many models perform poorly on 2022, and generally perform worse on more recent versions. (Bottom) All models perform very poorly at semantic changes, indicating an potential area for massive improvement. Most models perform well on function name changes and new feature (with the exception of Llama-3.2-3B, which is also the smallest model in this comparison).
• Figure 5: Comparison of sample and model differences. The left panel shows the distribution of sample difficulty, measured by the frequency of pass@10 scores across all models and seeds. The right panel presents the same distribution, but averaged for specific models across their seeds. Interestingly, individual models tend to exhibit bimodal distributions, indicating they are either consistently good or bad at specific problems. However, the aggregate distribution is less bimodal, suggesting that different models excel at different problems.