Analyzing Message-Code Inconsistency in AI Coding Agent-Authored Pull Requests

TL;DR

The paper investigates the reliability of AI-generated PR descriptions by introducing PR-MCI, a measure of message-code misalignment, and applying it to 23,247 Agentic-PRs from the AIDev dataset. It presents 974 manually annotated PRs to validate PR-MCI and derives a taxonomy of eight inconsistency types, with Phantom Changes dominating at 45.4%. The study finds that high-MCI PRs (1.7%) are associated with substantially worse outcomes, including a 51.7% drop in acceptance and a 3.5x increase in merge time, after controlling for confounders. These findings highlight the need for automated PR-MCI verification and improved generation processes to support trustworthy human-AI collaboration in software development.

Abstract

Pull request (PR) descriptions generated by AI coding agents are the primary channel for communicating code changes to human reviewers. However, the alignment between these messages and the actual changes remains unexplored, raising concerns about the trustworthiness of AI agents. To fill this gap, we analyzed 23,247 agentic PRs across five agents using PR message-code inconsistency (PR-MCI). We contributed 974 manually annotated PRs, found 406 PRs (1.7%) exhibited high PR-MCI, and identified eight PR-MCI types, revealing that descriptions claiming unimplemented changes was the most common issue (45.4%). Statistical tests confirmed that high-MCI PRs had 51.7% lower acceptance rates (28.3% vs. 80.0%) and took 3.5x longer to merge (55.8 vs. 16.0 hours). Our findings suggest that unreliable PR descriptions undermine trust in AI agents, highlighting the need for PR-MCI verification mechanisms and improved PR generation to enable trustworthy human-AI collaboration.

Figures (3)

• Figure 1: Methodology workflow for PR-MCI analysis.
• Figure 2: Heatmap of high-MCI prevalence (%) by agent$\times$task.
• Figure 3: Distribution of PR-MCI categories overall, by coding agent, and by task type.