Let's Make Every Pull Request Meaningful: An Empirical Analysis of Developer and Agentic Pull Requests
Haruhiko Yoshioka, Takahiro Monno, Haruka Tokumasu, Taiki Wakamatsu, Yuki Ota, Nimmi Weeraddana, Kenichi Matsumoto
TL;DR
The paper tackles why AI-generated PRs exhibit different merge outcomes by performing a large-scale, interpretable regression analysis on 40,214 PRs from the AIDev dataset, using 64 features across six families. It finds that submitter attributes largely determine merge probability for both human and agentic PRs, with human merges aided by reviews while agentic merges hinge more on code-change signals, and it reveals agent-specific patterns across multiple AI agents. The work provides actionable guidance for improving PR quality through human-AI collaboration and for selecting AI agents aligned with workflow goals. Overall, the study offers a nuanced view of how social and technical signals influence PR merging in mixed human-AI development environments.
Abstract
The automatic generation of pull requests (PRs) using AI agents has become increasingly common. Although AI-generated PRs are fast and easy to create, their merge rates have been reported to be lower than those created by humans. In this study, we conduct a large-scale empirical analysis of 40,214 PRs collected from the AIDev dataset. We extract 64 features across six families and fit statistical regression models to compare PR merge outcomes for human and agentic PRs, as well as across three AI agents. Our results show that submitter attributes dominate merge outcomes for both groups, while review-related features exhibit contrasting effects between human and agentic PRs. The findings of this study provide insights into improving PR quality through human-AI collaboration.