Understanding Dominant Themes in Reviewing Agentic AI-authored Code
Md. Asif Haider, Thomas Zimmermann
TL;DR
The paper probes how reviewers respond to agentic AI-authored code by deriving a 12-theme taxonomy of review comments through BERTopic clustering refined with GPT consolidation, and validating an open-source LLM (Gemma) as an annotation proxy against human labels. Using a curated AIDev subset, it demonstrates strong comment-level and PR-level annotation performance and reveals that reviewer attention centers on documentation, refactoring, styling, and functional correctness, with testing and security as key risk factors for rejection. Differences emerge between accepted and rejected PRs, notably with documentation and styling aiding acceptance, while security and build issues correlate with rejections. The findings offer practical guidance for improving agentic review processes, reducing noise, and shaping future data-driven training for automated review and repair systems.
Abstract
While prior work has examined the generation capabilities of Agentic AI systems, little is known about how reviewers respond to AI-authored code in practice. In this paper, we present a large-scale empirical study of code review dynamics in agent-generated PRs. Using a curated subset of the AIDev dataset, we analyze 19,450 inline review comments spanning 3,177 agent-authored PRs from real-world GitHub repositories. We first derive a taxonomy of 12 review comment themes using topic modeling combined with large language model (LLM)-assisted semantic clustering and consolidation. According to this taxonomy, we then investigate whether zero-shot prompts to LLM can reliably annotate review comments. Our evaluation against human annotations shows that open-source LLM achieves reasonably high exact match (78.63%), macro F1 score (0.78), and substantial agreement with human annotators at the review comment level. At the PR level, the LLM also correctly identifies the dominant review theme with 78% Top-1 accuracy and achieves an average Jaccard similarity of 0.76, indicating strong alignment with human judgments. Applying this annotation pipeline at scale, we find that apart from functional correctness and logical changes, reviews of agent-authored PRs predominantly focus on documentation gaps, refactoring needs, styling and formatting issues, with testing and security-related concerns. These findings suggest that while AI agents can accelerate code production, there remain gaps requiring targeted human review oversight.