Who Writes the Docs in SE 3.0? Agent vs. Human Documentation Pull Requests

TL;DR

The paper investigates how AI agents contribute to software documentation in SE3.0 by analyzing 1,997 documentation-related PRs from the AIDev dataset, comparing agent-authored versus human-authored changes. It finds that AI agents dominate documentation edits, with 66.1% of changed files edited by agents and 3.7% co-edited, and many agent edits are accepted with limited human follow-up. The study highlights risks for documentation quality assurance and human-AI collaboration, and provides a replication package to enable reproducibility and future studies. Overall, the work establishes a baseline understanding of agent-driven documentation in practice and motivates improved review practices and QA tooling for SE3.0 documentation workflows.

Abstract

As software engineering moves toward SE3.0, AI agents are increasingly used to carry out development tasks and contribute changes to software projects. It is therefore important to understand the extent of these contributions and how human developers review and intervene, since these factors shape the risks of delegating work to AI agents. While recent studies have examined how AI agents support software development tasks (e.g., code generation, issue resolution, and PR automation), their role in documentation tasks remains underexplored-even though documentation is widely consumed and shapes how developers understand and use software. Using the AIDev, we analyze 1,997 documentation-related pull requests (PRs) authored by AI agents and human developers, where documentation PRs are those that create or modify project documentation artifacts. We find that AI agents submit substantially more documentation-related PRs than humans in the studied repositories. We further observe that agent-authored documentation edits are typically integrated with little follow-up modification from humans, raising concerns about review practices and the reliability of agent-generated documentation. Overall, while AI agents already contribute substantially to documentation workflows, our results suggest concerns for emerging challenges for documentation quality assurance and human-AI collaboration in SE3.0.

Figures (5)

• Figure 1: Flowchart of Data Collection
• Figure 2: Number of docs-related PRs
• Figure 3: Breakdown of contributions by file
• Figure 4: Distribution of PRs categorized by the types of files
• Figure 5: Human Followup Impact