Human-Agent versus Human Pull Requests: A Testing-Focused Characterization and Comparison

TL;DR

This study provides the first large-scale, multi-language characterization of testing in PRs involving AI-based coding agents (HAPRs) versus human-only PRs (HPRs). By analyzing 6,582 HAPRs and 3,122 HPRs from the AIDev dataset across four languages and five agents, the authors quantify testing frequency, extent, context, and test smells. They find that while the likelihood of including tests is similar, HAPRs contribute substantially more test code during co-evolution, especially through adding new tests, yet overall test quality remains comparable based on test smells. The results offer practical guidance for integrating agents into testing work and point to future directions in maintaining and evolving test suites to avoid bloating, with a publicly available replication package for validation.

Abstract

AI-based coding agents are increasingly integrated into software development workflows, collaborating with developers to create pull requests (PRs). Despite their growing adoption, the role of human-agent collaboration in software testing remains poorly understood. This paper presents an empirical study of 6,582 human-agent PRs (HAPRs) and 3,122 human PRs (HPRs) from the AIDev dataset. We compare HAPRs and HPRs along three dimensions: (i) testing frequency and extent, (ii) types of testing-related changes (code-and-test co-evolution vs. test-focused), and (iii) testing quality, measured by test smells. Our findings reveal that, although the likelihood of including tests is comparable (42.9% for HAPRs vs. 40.0% for HPRs), HAPRs exhibit a larger extent of testing, nearly doubling the test-to-source line ratio found in HPRs. While test-focused task distributions are comparable, HAPRs are more likely to add new tests during co-evolution (OR=1.79), whereas HPRs prioritize modifying existing tests. Finally, although some test smell categories differ statistically, negligible effect sizes suggest no meaningful differences in quality. These insights provide the first characterization of how human-agent collaboration shapes testing practices.