Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development

Shyam Agarwal, Hao He, Bogdan Vasilescu

TL;DR

The paper investigates how autonomous, repository-level coding agents impact open-source software development, comparing them with IDE-based AI assistants. Using a staggered difference-in-differences design on the AIDev dataset and a Borusyak estimator, it shows that significant velocity gains occur primarily when agents are the first AI tool in a project, with diminished benefits once IDE-based AI use is established. However, across both AI-exposure groups, agent adoption consistently increases maintainability risks, including higher static-analysis warnings and cognitive complexity, signaling an enduring speed–quality trade-off. The findings underscore the need for provenance tracking, selective deployment, and proactive quality safeguards to maximize benefits while mitigating debt in AI-integrated development workflows.

Abstract

Large language model (LLM)-based coding agents increasingly act as autonomous contributors that generate and merge pull requests, yet their real-world effects on software projects are unclear, especially relative to widely adopted IDE-based AI assistants. We present a longitudinal causal study of agent adoption in open-source repositories using staggered difference-in-differences with matched controls. Using the AIDev dataset, we define adoption as the first agent-generated pull request and analyze monthly repository-level outcomes spanning development velocity (commits, lines added) and software quality (static-analysis warnings, cognitive complexity, duplication, and comment density). Results show large, front-loaded velocity gains only when agents are the first observable AI tool in a project; repositories with prior AI IDE usage experience minimal or short-lived throughput benefits. In contrast, quality risks are persistent across settings, with static-analysis warnings and cognitive complexity rising roughly 18% and 35%, indicating sustained agent-induced complexity debt even when velocity advantages fade. These heterogeneous effects suggest diminishing returns to AI assistance and highlight the need for quality safeguards, provenance tracking, and selective deployment of autonomous agents. Our findings establish an empirical basis for understanding how agentic and IDE-based tools interact, and motivate research on balancing acceleration with maintainability in AI-integrated development workflows.

AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development

TL;DR

The paper investigates how autonomous, repository-level coding agents impact open-source software development, comparing them with IDE-based AI assistants. Using a staggered difference-in-differences design on the AIDev dataset and a Borusyak estimator, it shows that significant velocity gains occur primarily when agents are the first AI tool in a project, with diminished benefits once IDE-based AI use is established. However, across both AI-exposure groups, agent adoption consistently increases maintainability risks, including higher static-analysis warnings and cognitive complexity, signaling an enduring speed–quality trade-off. The findings underscore the need for provenance tracking, selective deployment, and proactive quality safeguards to maximize benefits while mitigating debt in AI-integrated development workflows.

Abstract

Large language model (LLM)-based coding agents increasingly act as autonomous contributors that generate and merge pull requests, yet their real-world effects on software projects are unclear, especially relative to widely adopted IDE-based AI assistants. We present a longitudinal causal study of agent adoption in open-source repositories using staggered difference-in-differences with matched controls. Using the AIDev dataset, we define adoption as the first agent-generated pull request and analyze monthly repository-level outcomes spanning development velocity (commits, lines added) and software quality (static-analysis warnings, cognitive complexity, duplication, and comment density). Results show large, front-loaded velocity gains only when agents are the first observable AI tool in a project; repositories with prior AI IDE usage experience minimal or short-lived throughput benefits. In contrast, quality risks are persistent across settings, with static-analysis warnings and cognitive complexity rising roughly 18% and 35%, indicating sustained agent-induced complexity debt even when velocity advantages fade. These heterogeneous effects suggest diminishing returns to AI assistance and highlight the need for quality safeguards, provenance tracking, and selective deployment of autonomous agents. Our findings establish an empirical basis for understanding how agentic and IDE-based tools interact, and motivate research on balancing acceleration with maintainability in AI-integrated development workflows.
Paper Structure (11 sections, 2 figures, 2 tables)

This paper contains 11 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Data Collection and Treatment Definition
  5. Outcomes and Covariates
  6. Causal Inference Strategy
  7. Results and Discussion
  8. Development Velocity
  9. Software Quality
  10. Implications and Ethical considerations
  11. Conclusion

Figures (2)

  • Figure 1: Monthly distribution of agent adoption dates, separated by prior AI exposure. Most adoptions occur between April and June 2025, with AF repositories more prevalent overall.
  • Figure 2: Estimated post-adoption effects of agentic coding tools by prior AI exposure. AF repositories gain velocity and accumulate maintainability risks; IF repositories show minimal velocity gains but comparable maintainability increases.