Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub

TL;DR

This paper tackles the problem of understanding how autonomous AI teammates, operating via agentic pull requests (PRs), affect software security in real-world development. It employs a large-scale, mixed-methods approach on the AIDev dataset to identify security-relevant Agentic-PRs, quantify their prevalence, and examine outcomes across agents, languages, and code-change types, complemented by open coding to reveal recurring security actions and intents. The study finds that security-related Agentic-PRs are meaningful yet a minority of activity, with security work often embedded in refactoring, testing, and hardening efforts; reviewers scrutinize these changes more heavily, resulting in longer review times and lower merge rates, with rejection more tied to complexity than to explicit security topics. The work offers design insights for review-aware secure AI teammates, such as favoring scoped, well-explained security changes and tailoring behavior to ecosystem norms, thereby informing governance and safety practices in GenAI-enabled software development.

Abstract

Autonomous coding agents are increasingly deployed as AI teammates in modern software engineering, independently authoring pull requests (PRs) that modify production code at scale. This study aims to systematically characterize how autonomous coding agents contribute to software security in practice, how these security-related contributions are reviewed and accepted, and which observable signals are associated with PR rejection. We conduct a large-scale empirical analysis of agent-authored PRs using the AIDev dataset, comprising of over 33,000 curated PRs from popular GitHub repositories. Security-relevant PRs are identified using a keyword filtering strategy, followed by manual validation, resulting in 1,293 confirmed security-related agentic-PRs. We then analyze prevalence, acceptance outcomes, and review latency across autonomous agents, programming ecosystems, and types of code changes. Moreover, we apply qualitative open coding to identify recurring security-related actions and underlying intents, and examine review metadata to identify early signals associated with PR rejection. Security-related Agentic-PRs constitute a meaningful share of agent activity (approximately 4\%). Rather than focusing solely on narrow vulnerability fixes, agents most frequently perform supportive security hardening activities, including testing, documentation, configuration, and improved error handling. Compared to non-security PRs, security-related Agentic-PRs exhibit lower merge rates and longer review latency, reflecting heightened human scrutiny, with variation across agents and programming ecosystems. PR rejection is more strongly associated with PR complexity and verbosity than with explicit security topics.

Figures (6)

• Figure 1: Overview of our Study's Methodology.
• Figure 2: Examples of open coding applied to security-related Agentic pull requests.
• Figure 3: RQ1 Results -- Prevalence of Security-relevant PRs Across Different Autonomous Coding Agents
• Figure 4: RQ2 Results -- Outcomes of Security-Related Pull Requests Authored by Agents.
• Figure 5: Co-occurrence Network of Security Actions.