When AI Agents Touch CI/CD Configurations: Frequency and Success
Taher A. Ghaleb
TL;DR
This paper addresses how AI agents interact with CI/CD configurations, focusing on YAML-based changes across a large, real-world dataset. Using the AIDev-pop resource and GitHub Actions data, the authors quantify YAML modification frequency, platform concentration, and downstream PR and build outcomes, applying chi-square tests and agent-level analyses. They show that AI agents modify CI/CD configurations sparingly (3.25%), concentrate on GitHub Actions (96.77%), and generally produce reliable changes, with Copilot displaying a notable CI/CD specialization that yields higher merge and build-success rates. The findings highlight platform biases, the potential for configuration-focused agent capabilities, and the need for cross-platform training and validation to improve AI-assisted DevOps in diverse CI/CD ecosystems.
Abstract
AI agents are increasingly used in software development, yet their interaction with CI/CD configurations is not well studied. We analyze 8,031 agentic pull requests (PRs) from 1,605 GitHub repositories where AI agents touch YAML configurations. CI/CD configuration files account for 3.25% of agent changes, varying by agent (Devin: 4.83%, Codex: 2.01%, p < 0.001). When agents modify CI/CD, 96.77% target GitHub Actions. Agentic PRs with CI/CD changes merge slightly less often than others (67.77% vs. 71.80%), except for Copilot, whose CI/CD changes merge 15.63 percentage points more often. Across 99,930 workflow runs, build success rates are comparable for CI/CD and non-CI/CD changes (75.59% vs. 74.87%), though three agents show significantly higher success when modifying CI/CD. These results show that AI agents rarely modify CI/CD and focus mostly on GitHub Actions, yet their configuration changes are as reliable as regular code. Copilot's strong CI/CD performance despite lower acceptance suggests emerging configuration specialization, with implications for agent training and DevOps automation.