Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requests
Dung Pham, Taher A. Ghaleb
TL;DR
This paper compares agentic pull requests (APRs) to human pull requests (HPRs) using a large-scale dataset of 33,596 APRs and 6,618 HPRs across multiple coding agents. It introduces alignment metrics for PR–commit, patch–commit, and an LLM-based consistency score to evaluate description quality at both commit and PR levels. The study finds that APRs produce smaller, more task-focused changes with faster symbol churn and lower merge rates, while humans handle broader PR narratives more effectively; commit messages are stronger for APRs, but overall PR-level descriptions lag. These results illuminate a gap between micro-level precision and macro-level communication in agent-driven development and suggest targeted directions to improve agent reasoning over full changesets and to enhance PR descriptions for reliability and maintainability.
Abstract
AI coding agents can autonomously generate pull requests (PRs), yet little is known about how their contributions compare to those of humans. We analyze 33,596 agent-generated PRs (APRs) and 6,618 human PRs (HPRs) to compare code-change characteristics and message quality. We observe that APR-introduced symbols (functions and classes) are removed much sooner than those in HPRs (median time to removal 3 vs. 34 days) and are also removed more often (symbol churn 7.33% vs. 4.10%), reflecting a focus on other tasks like documentation and test updates. Agents generate stronger commit-level messages (semantic similarity 0.72 vs. 0.68) but lag humans at PR-level summarization (PR-commit similarity 0.86 vs. 0.88). Commit message length is the best predictor of description quality, indicating reliance on individual commits over full-PR reasoning. These findings highlight a gap between agents' micro-level precision and macro-level communication, suggesting opportunities to improve agent-driven development workflows.