Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents

Jai Lal Lulla, Seyedmoein Mohsenimofidi, Matthias Galster, Jie M. Zhang, Sebastian Baltes, Christoph Treude

TL;DR

The paper investigates how repository-level AGENTS.md files shape the efficiency of autonomous AI coding agents on real GitHub pull requests. Using a paired within-repo design, it compares agent runs with and without AGENTS.md, focusing on token usage and wall-clock time as efficiency metrics. The study finds that AGENTS.md presence reduces mean output tokens by about 20% and median time-to-completion by about 28–29%, while maintaining comparable task completion behavior, highlighting practical cost and latency benefits. These findings suggest that repository-level instructions can materially influence agent behavior and efficiency in software development workflows, and they motivate broader, multi-agent and correctness-focused follow-up research.

Abstract

AI coding agents such as Codex and Claude Code are increasingly used to autonomously contribute to software repositories. However, little is known about how repository-level configuration artifacts affect operational efficiency of the agents. In this paper, we study the impact of AGENTS$.$md files on the runtime and token consumption of AI coding agents operating on GitHub pull requests. We analyze 10 repositories and 124 pull requests, executing agents under two conditions: with and without an AGENTS$.$md file. We measure wall-clock execution time and token usage during agent execution. Our results show that the presence of AGENTS$.$md is associated with a lower median runtime ($Δ28.64$%) and reduced output token consumption ($Δ16.58$%), while maintaining a comparable task completion behavior. Based on these results, we discuss immediate implications for the configuration and deployment of AI coding agents in practice, and outline a broader research agenda on the role of repository-level instructions in shaping the behavior, efficiency, and integration of AI coding agents in software development workflows.

On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents

TL;DR

The paper investigates how repository-level AGENTS.md files shape the efficiency of autonomous AI coding agents on real GitHub pull requests. Using a paired within-repo design, it compares agent runs with and without AGENTS.md, focusing on token usage and wall-clock time as efficiency metrics. The study finds that AGENTS.md presence reduces mean output tokens by about 20% and median time-to-completion by about 28–29%, while maintaining comparable task completion behavior, highlighting practical cost and latency benefits. These findings suggest that repository-level instructions can materially influence agent behavior and efficiency in software development workflows, and they motivate broader, multi-agent and correctness-focused follow-up research.

Abstract

AI coding agents such as Codex and Claude Code are increasingly used to autonomously contribute to software repositories. However, little is known about how repository-level configuration artifacts affect operational efficiency of the agents. In this paper, we study the impact of AGENTSmd files on the runtime and token consumption of AI coding agents operating on GitHub pull requests. We analyze 10 repositories and 124 pull requests, executing agents under two conditions: with and without an AGENTSmd file. We measure wall-clock execution time and token usage during agent execution. Our results show that the presence of AGENTSmd is associated with a lower median runtime (%) and reduced output token consumption (%), while maintaining a comparable task completion behavior. Based on these results, we discuss immediate implications for the configuration and deployment of AI coding agents in practice, and outline a broader research agenda on the role of repository-level instructions in shaping the behavior, efficiency, and integration of AI coding agents in software development workflows.
Paper Structure (16 sections, 1 figure, 1 table)

This paper contains 16 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Study Design
  4. Data Collection and Analysis
  5. Agent Selection
  6. Repository Sampling and Inclusion Criteria
  7. Pull Request Selection and Task Construction
  8. Reconstructing Pre-PR Repository State
  9. Issue Generation for PRs Lacking Usable Descriptions
  10. Experimental Conditions
  11. Running the Experiments
  12. Metrics
  13. Results
  14. Token usage
  15. Wall-clock time-to-completion
  16. ...and 1 more sections

Figures (1)

  • Figure 1: Wall-clock time-to-completion distributions for agent runs with and without AGENTS.md