Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration

Jialong Chen, Xander Xu, Hu Wei, Chuan Chen, Bing Zhao

TL;DR

SWE-CI is proposed, the first repository-level benchmark built upon the Continuous Integration loop, aiming to shift the evaluation paradigm for code generation from static, short-term functional correctness toward dynamic, long-term maintainability.

Abstract

Large language model (LLM)-powered agents have demonstrated strong capabilities in automating software engineering tasks such as static bug fixing, as evidenced by benchmarks like SWE-bench. However, in the real world, the development of mature software is typically predicated on complex requirement changes and long-term feature iterations -- a process that static, one-shot repair paradigms fail to capture. To bridge this gap, we propose \textbf{SWE-CI}, the first repository-level benchmark built upon the Continuous Integration loop, aiming to shift the evaluation paradigm for code generation from static, short-term \textit{functional correctness} toward dynamic, long-term \textit{maintainability}. The benchmark comprises 100 tasks, each corresponding on average to an evolution history spanning 233 days and 71 consecutive commits in a real-world code repository. SWE-CI requires agents to systematically resolve these tasks through dozens of rounds of analysis and coding iterations. SWE-CI provides valuable insights into how well agents can sustain code quality throughout long-term evolution.

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration

TL;DR

SWE-CI is proposed, the first repository-level benchmark built upon the Continuous Integration loop, aiming to shift the evaluation paradigm for code generation from static, short-term functional correctness toward dynamic, long-term maintainability.

Abstract

Large language model (LLM)-powered agents have demonstrated strong capabilities in automating software engineering tasks such as static bug fixing, as evidenced by benchmarks like SWE-bench. However, in the real world, the development of mature software is typically predicated on complex requirement changes and long-term feature iterations -- a process that static, one-shot repair paradigms fail to capture. To bridge this gap, we propose \textbf{SWE-CI}, the first repository-level benchmark built upon the Continuous Integration loop, aiming to shift the evaluation paradigm for code generation from static, short-term \textit{functional correctness} toward dynamic, long-term \textit{maintainability}. The benchmark comprises 100 tasks, each corresponding on average to an evolution history spanning 233 days and 71 consecutive commits in a real-world code repository. SWE-CI requires agents to systematically resolve these tasks through dozens of rounds of analysis and coding iterations. SWE-CI provides valuable insights into how well agents can sustain code quality throughout long-term evolution.
Paper Structure (21 sections, 3 equations, 6 figures)

This paper contains 21 sections, 3 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Measuring the Agent's Ability to Maintain Codebase
  3. Task formalization
  4. Normalized Change
  5. EvoScore
  6. SWE-CI
  7. Data curation
  8. Step 1: Repository Collection.
  9. Step 2: Commit Span Extraction.
  10. Step 3: Environment Construction.
  11. Step 4: Case Filtering.
  12. Dual-agent evaluation protocol
  13. Architect agent.
  14. Programmer.
  15. Experiments
  16. ...and 6 more sections

Figures (6)

  • Figure 1: Unlike previous benchmarks, SWE-CI proposes an evolution-based evaluation. The red and blue arrows represent the actions of functions $\mathsf{require}$ and $\mathsf{code}$, respectively. Dashed lines indicate processes that are unknown to the user.
  • Figure 2: Data curation process of SWE-CI.
  • Figure 3: SWE-CI uses an architect-programmer dual-agent workflow to model the continuous integration cycle of professional software teams in the real world.
  • Figure 4: The EvoScore variation of state-of-the-art models from 8 providers on SWE-CI.
  • Figure 5: The model's EvoScore ranking changes as $\gamma$ increases. When $\gamma > 1$, higher-ranking models indicate better codebase maintenance.
  • ...and 1 more figures