How Do Agents Perform Code Optimization? An Empirical Study

TL;DR

The paper addresses how AI agents perform code optimization in real-world software, performing the first empirical comparison of agent- and human-authored perf PRs using 407 perf PRs from the AIDev dataset. It adopts an extended SysLLMatic catalog to label optimization patterns and analyzes performance validation practices, finding that AI-generated perf PRs largely reuse human-like optimization patterns but provide weaker validation, often relying on static reasoning rather than benchmarking. The study introduces 14 new patterns to capture real-world changes and reports that agent PRs merge faster but with lower validation frequency, highlighting a critical gap in empirical validation for agent-driven optimization. The findings inform future directions for integrating automated benchmarking, profiling, and correctness checks to enable safer, broader exploration of performance optimizations by AI agents.

Abstract

Performance optimization is a critical yet challenging aspect of software development, often requiring a deep understanding of system behavior, algorithmic tradeoffs, and careful code modifications. Although recent advances in AI coding agents have accelerated code generation and bug fixing, little is known about how these agents perform on real-world performance optimization tasks. We present the first empirical study comparing agent- and human-authored performance optimization commits, analyzing 324 agent-generated and 83 human-authored PRs from the AIDev dataset across adoption, maintainability, optimization patterns, and validation practices. We find that AI-authored performance PRs are less likely to include explicit performance validation than human-authored PRs (45.7\% vs. 63.6\%, $p=0.007$). In addition, AI-authored PRs largely use the same optimization patterns as humans. We further discuss limitations and opportunities for advancing agentic code optimization.

Figures (4)

• Figure 1: Methodology overview illustrating the data collection pipeline and how the resulting dataset supports answering RQ1 and RQ2.
• Figure 2:
• Figure 3: Distribution of high-level optimization patterns in AI- and human-authored perf PRs. Both groups exhibit similar distributions, with memory and data-locality optimizations and algorithm optimizations dominating.
• Figure 4: Distribution of types of validation reported in perf PRs. Agent PRs rely predominantly on static reasoning checks, whereas human PRs more frequently report benchmark-based validation in addition to static reasoning.