Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle

Yuheng Tang, Kaijie Zhu, Bonan Ruan, Chuqi Zhang, Michael Yang, Hongwei Li, Suyue Guo, Tianneng Shi, Zekun Li, Christopher Kruegel, Giovanni Vigna, Dawn Song, William Yang Wang, Lun Wang, Yangruibo Ding, Zhenkai Liang, Wenbo Guo

TL;DR

DevOps-Gym addresses a key gap by providing the first end-to-end benchmark for evaluating AI agents across the complete DevOps cycle, moving beyond isolated code tasks to dynamic, tool-augmented workflows in real-world Java and Go projects. The framework covers four stages—build/configuration, monitoring, issue resolving, and test generation—across 704 validated tasks and 18 end-to-end pipelines, with standardized tool interfaces and rigorous data curation to prevent contamination. Empirical evaluation shows substantial limitations in current agentic systems, especially in monitoring and cross-language build tasks, and reveals that even state-of-the-art models struggle to perform end-to-end DevOps workflows. The work highlights the need for advances in multi-step planning, tool reasoning, and temporal/dynamic reasoning to realize AI-assisted automation of the full DevOps lifecycle, offering a path for future benchmarks and method development.

Abstract

Even though demonstrating extraordinary capabilities in code generation and software issue resolving, AI agents' capabilities in the full software DevOps cycle are still unknown. Different from pure code generation, handling the DevOps cycle in real-world software, including developing, deploying, and managing, requires analyzing large-scale projects, understanding dynamic program behaviors, leveraging domain-specific tools, and making sequential decisions. However, existing benchmarks focus on isolated problems and lack environments and tool interfaces for DevOps. We introduce DevOps-Gym, the first end-to-end benchmark for evaluating AI agents across core DevOps workflows: build and configuration, monitoring, issue resolving, and test generation. DevOps-Gym includes 700+ real-world tasks collected from 30+ projects in Java and Go. We develop a semi-automated data collection mechanism with rigorous and non-trivial expert efforts in ensuring the task coverage and quality. Our evaluation of state-of-the-art models and agents reveals fundamental limitations: they struggle with issue resolving and test generation in Java and Go, and remain unable to handle new tasks such as monitoring and build and configuration. These results highlight the need for essential research in automating the full DevOps cycle with AI agents.

DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle

TL;DR

DevOps-Gym addresses a key gap by providing the first end-to-end benchmark for evaluating AI agents across the complete DevOps cycle, moving beyond isolated code tasks to dynamic, tool-augmented workflows in real-world Java and Go projects. The framework covers four stages—build/configuration, monitoring, issue resolving, and test generation—across 704 validated tasks and 18 end-to-end pipelines, with standardized tool interfaces and rigorous data curation to prevent contamination. Empirical evaluation shows substantial limitations in current agentic systems, especially in monitoring and cross-language build tasks, and reveals that even state-of-the-art models struggle to perform end-to-end DevOps workflows. The work highlights the need for advances in multi-step planning, tool reasoning, and temporal/dynamic reasoning to realize AI-assisted automation of the full DevOps lifecycle, offering a path for future benchmarks and method development.

Abstract

Even though demonstrating extraordinary capabilities in code generation and software issue resolving, AI agents' capabilities in the full software DevOps cycle are still unknown. Different from pure code generation, handling the DevOps cycle in real-world software, including developing, deploying, and managing, requires analyzing large-scale projects, understanding dynamic program behaviors, leveraging domain-specific tools, and making sequential decisions. However, existing benchmarks focus on isolated problems and lack environments and tool interfaces for DevOps. We introduce DevOps-Gym, the first end-to-end benchmark for evaluating AI agents across core DevOps workflows: build and configuration, monitoring, issue resolving, and test generation. DevOps-Gym includes 700+ real-world tasks collected from 30+ projects in Java and Go. We develop a semi-automated data collection mechanism with rigorous and non-trivial expert efforts in ensuring the task coverage and quality. Our evaluation of state-of-the-art models and agents reveals fundamental limitations: they struggle with issue resolving and test generation in Java and Go, and remain unable to handle new tasks such as monitoring and build and configuration. These results highlight the need for essential research in automating the full DevOps cycle with AI agents.
Paper Structure (21 sections, 3 figures, 6 tables)

This paper contains 21 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Construction of DevOps-Gym
  4. Overview
  5. Task Construction for Build and Configuration
  6. Task Construction for Monitoring
  7. Task Construction for Issue Resolving and Test Generation
  8. End-to-end Tasks
  9. Evaluation
  10. Experimental Setup
  11. Results
  12. Stability of Agent Performance Across Multiple Independent Runs
  13. Conclusion and Future Works
  14. Clarification on the Use of LLMs
  15. Dataset Contamination Prevention
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Overview of DevOps-Gym. It includes four core stages of DevOps: Build & Configuration, Monitoring, Issue Resolving, and Test Generation. Each stage requires an AI agent to leverage a distinct set of command-line tools to solve realistic tasks.
  • Figure 2: Performance comparison across different agentic frameworks and LLMs.
  • Figure 3: Monitoring anomaly distribution.