How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests

TL;DR

The paper addresses how AI coding agents alter code via pull requests and how well agent descriptions reflect changes. It analyzes 24,014 Agentic PRs versus 5,081 Human PRs from the MSR 2026 AIDev dataset, reconstructing commit histories and applying structural metrics along with lexical and semantic description-diff similarity analyses using TF-IDF, BM25, CodeBERT, and GraphCodeBERT. The key findings show Agentic PRs enact more compact, localized edits with fewer commits and touched files, while human PRs are broader; nonetheless, agent descriptions align with edits at least as well as human descriptions, with high semantic similarity. These results reveal distinctive yet coherent AI-driven contribution patterns and offer practical implications for PR review, changelog generation, and automated quality checks, while highlighting avenues for future work on maintainability and defect risk as coding agents evolve.

Abstract

AI coding agents are increasingly acting as autonomous contributors by generating and submitting pull requests (PRs). However, we lack empirical evidence on how these agent-generated PRs differ from human contributions, particularly in how they modify code and describe their changes. Understanding these differences is essential for assessing their reliability and impact on development workflows. Using the MSR 2026 Mining Challenge version of the AIDev dataset, we analyze 24,014 merged Agentic PRs (440,295 commits) and 5,081 merged Human PRs (23,242 commits). We examine additions, deletions, commits, and files touched, and evaluate the consistency between PR descriptions and their diffs using lexical and semantic similarity. Agentic PRs differ substantially from Human PRs in commit count (Cliff's $δ= 0.5429$) and show moderate differences in files touched and deleted lines. They also exhibit slightly higher description-to-diff similarity across all measures. These findings provide a large-scale empirical characterization of how AI coding agents contribute to open source development.

Figures (5)

• Figure 1: Four-step workflow: dataset collection, commit-data extension, PR filtering, and structural and similarity analysis.
• Figure 2: Distribution of LOC added and deleted in agentic vs. human PRs.
• Figure 3: Distribution of commits and files touched across agents and human PRs.
• Figure 4: Kernel density estimates comparing lexical and semantic similarity distributions.
• Figure 5: Distribution of similarity scores across lexical and semantic metrics for both Agentic and Human PRs.