Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution

KN Ajay Shastry, Ganesh Senrayan, Shrey Satapara, Pranoy Panda, Chaitanya Devaguptapu

Abstract

Existing datasets for coding agents evaluate performance on isolated, single pull request (PR) tasks in a stateless manner, failing to capture the reality of real-world software development where code changes accumulate, technical debt accrues, and test suites grow over time. To bridge this gap, we introduce an automated coding task generation framework, which helps generate our dataset SWE-STEPS, that evaluates coding agents on long-horizon tasks through two realistic settings mirroring actual developer workflows: Conversational coding with iterative requests, and single-shot Project Requirement document (PRD)-based coding. Unlike existing datasets that evaluate agents on disjointed Pull Requests (PRs), our framework assesses performance across chains of dependent PRs, enabling evaluation of sequential execution, regression verification, and long-term repository health. We discover that widely used isolated PR evaluations yield inflated success rates, w.r.t. our settings - overshooting performance by as much as 20 percentage points - because they ignore the ``spillover'' effects of previous inefficient or buggy code. Furthermore, our analysis reveals that even when agents successfully resolve issues, they degrade repository health by generating code with higher cognitive complexity and technical debt compared to human developers, underscoring the necessity for multidimensional evaluation.

Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution

Abstract

Existing datasets for coding agents evaluate performance on isolated, single pull request (PR) tasks in a stateless manner, failing to capture the reality of real-world software development where code changes accumulate, technical debt accrues, and test suites grow over time. To bridge this gap, we introduce an automated coding task generation framework, which helps generate our dataset SWE-STEPS, that evaluates coding agents on long-horizon tasks through two realistic settings mirroring actual developer workflows: Conversational coding with iterative requests, and single-shot Project Requirement document (PRD)-based coding. Unlike existing datasets that evaluate agents on disjointed Pull Requests (PRs), our framework assesses performance across chains of dependent PRs, enabling evaluation of sequential execution, regression verification, and long-term repository health. We discover that widely used isolated PR evaluations yield inflated success rates, w.r.t. our settings - overshooting performance by as much as 20 percentage points - because they ignore the ``spillover'' effects of previous inefficient or buggy code. Furthermore, our analysis reveals that even when agents successfully resolve issues, they degrade repository health by generating code with higher cognitive complexity and technical debt compared to human developers, underscoring the necessity for multidimensional evaluation.

Paper Structure

This paper contains 27 sections, 10 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Design and Construction
  4. Dataset Desiderata & Framework Formulation
  5. Proposed Data Generation Framework
  6. SWE-STEPS Dataset
  7. Experimental Setup
  8. Agent Configuration
  9. Environment and Execution Protocol
  10. Results And Analysis
  11. Conclusion
  12. Impact Statement
  13. Appendix
  14. Dataset and Execution Environment
  15. Dataset Composition and Attributes
  16. ...and 12 more sections

Figures (10)

  • Figure 1: Task Extraction Pipeline. Tasks are extracted from PRs and verified via the execution sandbox.
  • Figure 2: Distribution of tasks across repositories
  • Figure 3: Comparative Analysis of Code Quality Evolution. The figure illustrates the progression of $\Delta$ Cognitive Complexity across conan, haystack, and moto. Plots reveal that autonomous agents tend to introduce higher complexity compared to the human code.
  • Figure 4: The agent conflates the current sub-task with artifacts found in the environment from a previous sub-task.
  • Figure 5: Repository task lists distribution (pie charts)
  • ...and 5 more figures