Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality

Anwar Ghammam, Mohamed Almukhtar

TL;DR

This work tackles the quality of AI-generated build code by providing the first empirical study of AI-produced build-script changes across Maven, Gradle, CMake, and Make using the AIDev dataset. It combines Sniffer-based smell detection with manual labeling to quantify how AI-generated changes introduce or remove smells and to assess developer acceptance of Agentic-PRs. The findings show that AI agents can both introduce and remove build smells, yet a majority of agent-generated PRs are merged with limited human intervention, signaling growing trust in automated build changes. The paper delivers an open, labeled dataset and actionable insights to guide smell-aware generation and governance of AI-built build scripts, paving the way for automated quality controls in build pipelines.

Abstract

The rapid adoption of AI coding agents for software development has raised important questions about the quality and maintainability of the code they produce. While prior studies have examined AI-generated source code, the impact of AI coding agents on build systems-a critical yet understudied component of the software lifecycle-remains largely unexplored. This data mining challenge focuses on AIDev, the first large-scale, openly available dataset capturing agent-authored pull requests (Agentic-PRs) from real-world GitHub repositories. Our paper leverages this dataset to investigate (RQ1) whether AI coding agents generate build code with quality issues (e.g., code smells), (RQ2) to what extent AI agents can eliminate code smells from build code, and (RQ3) to what extent Agentic-PRs are accepted by developers. We identified 364 maintainability and security-related build smells across varying severity levels, indicating that AI-generated build code can introduce quality issues-such as lack of error handling, and hardcoded paths or URLs-while also, in some cases, removing existing smells through refactorings (e.g., Pull Up Module and Externalize Properties). Notably, more than 61\% of Agentic-PRs are approved and merged with minimal human intervention. This dual impact underscores the need for future research on AI-aware build code quality assessment to systematically evaluate, guide, and govern AI-generated build systems code.

AI builds, We Analyze: An Empirical Study of AI-Generated Build Code Quality

TL;DR

This work tackles the quality of AI-generated build code by providing the first empirical study of AI-produced build-script changes across Maven, Gradle, CMake, and Make using the AIDev dataset. It combines Sniffer-based smell detection with manual labeling to quantify how AI-generated changes introduce or remove smells and to assess developer acceptance of Agentic-PRs. The findings show that AI agents can both introduce and remove build smells, yet a majority of agent-generated PRs are merged with limited human intervention, signaling growing trust in automated build changes. The paper delivers an open, labeled dataset and actionable insights to guide smell-aware generation and governance of AI-built build scripts, paving the way for automated quality controls in build pipelines.

Abstract

The rapid adoption of AI coding agents for software development has raised important questions about the quality and maintainability of the code they produce. While prior studies have examined AI-generated source code, the impact of AI coding agents on build systems-a critical yet understudied component of the software lifecycle-remains largely unexplored. This data mining challenge focuses on AIDev, the first large-scale, openly available dataset capturing agent-authored pull requests (Agentic-PRs) from real-world GitHub repositories. Our paper leverages this dataset to investigate (RQ1) whether AI coding agents generate build code with quality issues (e.g., code smells), (RQ2) to what extent AI agents can eliminate code smells from build code, and (RQ3) to what extent Agentic-PRs are accepted by developers. We identified 364 maintainability and security-related build smells across varying severity levels, indicating that AI-generated build code can introduce quality issues-such as lack of error handling, and hardcoded paths or URLs-while also, in some cases, removing existing smells through refactorings (e.g., Pull Up Module and Externalize Properties). Notably, more than 61\% of Agentic-PRs are approved and merged with minimal human intervention. This dual impact underscores the need for future research on AI-aware build code quality assessment to systematically evaluate, guide, and govern AI-generated build systems code.
Paper Structure (21 sections, 1 figure, 1 table)

This paper contains 21 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Build Systems
  4. Code Smells & Refactoring
  5. Sniffer: A Static Analyser for build code
  6. Related Works
  7. Build Maintenance
  8. AI-based Code Generation
  9. Methodology
  10. Dataset
  11. Data Preprocessing
  12. Build Code Quality Assessment With Sniffer
  13. Qualitative Analysis via Manual Labeling
  14. Results
  15. RQ1: AI-generated Build code Quality
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Introduced Code Smells per AI Agent