A Task-Level Evaluation of AI Agents in Open-Source Projects

TL;DR

The paper tackles the problem of evaluating autonomous coding agents in real-world open-source workflows across task types. It introduces a task-aware evaluation framework using the AIDev dataset to compare five agents on PR acceptance, review discussion, and commit message quality, with a C-Good classifier assessing commit quality. Key findings show Codex achieves the highest PR acceptance but low commit-quality, Copilot drives the most review activity, while Claude and Cursor deliver higher-quality commit messages with notable task variability. The results highlight that different evaluation dimensions capture complementary aspects of agent performance and inform practical tool selection and future agent design for collaborative software engineering.

Abstract

In this paper, we present a comparative study of five autonomous coding agents using AIDev-pop, which is a public dataset containing thousands of AI-generated pull requests (PRs) across popular open-source repositories. We evaluate agents' performance along three task-aware dimensions spanning the PR lifecycle: (1) PR acceptance rate, (2) review discussion volume, and (3) commit message quality. Our quantitative analysis finds that Codex consistently achieves high PR acceptance rates across most task categories, while Copilot's PRs trigger the highest volume of both human and automated review discussions. In contrast, commit-level quality varies independently of acceptance outcomes. Claude and Cursor produce higher proportions of high-quality commit messages across several task types, and Codex exhibiting comparatively lower commit quality despite strong integration outcomes. Our findings inform selection and improvements of AI agents for their effective integration to collaborative software engineering.