Context Engineering for AI Agents in Open-Source Software

TL;DR

The paper investigates how AI agents in open-source software receive project-specific context through AI configuration files, focusing on AGENTS.md. Using mining of 10,000 GitHub repositories and qualitative analysis of 155 AGENTS.md files, it documents wide variation in content, structure, and writing style, and traces evolution through 169 commits in 10 files. It shows no single standard yet and argues that AI configuration files behave as maintained software artifacts that can be versioned and tested. The study highlights opportunities to improve interoperability with standard schemas and to study how structure and style influence agent outputs, positioning repositories as real-world laboratories for prompt and context engineering.

Abstract

GenAI-based coding assistants have disrupted software development. Their next generation is agent-based, operating with more autonomy and potentially without human oversight. One challenge is to provide AI agents with sufficient context about the software projects they operate in. Like humans, AI agents require contextual information to develop solutions that are in line with the target architecture, interface specifications, coding guidelines, standard workflows, and other project-specific policies. Popular AI agents for software development (e.g., Claude Code) advocate for maintaining tool-specific version-controlled Markdown files that cover aspects such as the project structure, building and testing, or code style. The content of these files is automatically added to each prompt. AGENTS$.$md has emerged as a potential standard that consolidates tool-specific formats. However, little is known about whether and how developers adopt this format. Therefore, in this paper, we present the results of a preliminary study investigating the adoption of AI configuration files in 466 open-source software projects, what information developers provide in these files, how they present that information, and how the files evolve over time. Our findings indicate that there is no established structure yet, and that there is a lot of variation in terms of how context is provided (descriptive, prescriptive, prohibitive, explanatory, conditional). We see great potential in studying which modifications in structure or presentation can positively affect the quality of the generated content. Finally, our analysis of commits modifying AGENTS$.$md files provides first insights into how projects continuously extend and maintain these files. We conclude the paper by outlining how the adoption of AI configuration files provides a unique opportunity to study real-world prompt and context engineering.

Figures (1)

• Figure 1: Data collection process.