Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

APEX-SWE

Abhi Kottamasu, Akul Datta, Aakash Barthwal, Chirag Mahapatra, Ajay Arun, Adarsh Hiremath, Brendan Foody, Bertie Vidgen

TL;DR

APEX-SWE targets real-world software engineering by evaluating frontier AI models on end-to-end Integration and Observability tasks, going beyond function-level benchmarks. The study shows that success hinges on epistemic reasoning and deliberate verification, not mere code generation, with Gemini 3 Pro achieving the best Pass@1 at $25\%$. The open-source dev set and harness enable reproducibility, and the findings underscore the need for systems that extract strict specifications, iteratively debug mental models, and validate outcomes against empirical states. The work highlights practical implications for deploying AI in production software, where robust integration and reliable debugging capabilities are essential for scalable AI-assisted engineering.

Abstract

We introduce the AI Productivity Index for Software Engineering (APEX-SWE), a benchmark for assessing whether frontier AI models can execute economically valuable software engineering work. Unlike existing evaluations that focus on narrow, well-defined tasks, APEX-SWE assesses two novel task types that reflect real-world software engineering work: (1) Integration tasks (n=100), which require constructing end-to-end systems across heterogeneous cloud primitives, business applications, and infrastructure-as-code services, and (2) Observability tasks (n=100), which require debugging production failures using telemetry signals such as logs and dashboards, as well as unstructured context. We evaluated eight frontier models on APEX-SWE. Gemini 3 Pro (Thinking = High) performs best, with a Pass@1 score of 25\%. Our analysis shows that strong performance is primarily driven by epistemic reasoning, defined as the ability to distinguish between assumptions and verified facts, combined with agency to resolve uncertainty prior to acting. We open-source the APEX-SWE evaluation harness and a dev set (n=50).

APEX-SWE

TL;DR

APEX-SWE targets real-world software engineering by evaluating frontier AI models on end-to-end Integration and Observability tasks, going beyond function-level benchmarks. The study shows that success hinges on epistemic reasoning and deliberate verification, not mere code generation, with Gemini 3 Pro achieving the best Pass@1 at . The open-source dev set and harness enable reproducibility, and the findings underscore the need for systems that extract strict specifications, iteratively debug mental models, and validate outcomes against empirical states. The work highlights practical implications for deploying AI in production software, where robust integration and reliable debugging capabilities are essential for scalable AI-assisted engineering.

Abstract

We introduce the AI Productivity Index for Software Engineering (APEX-SWE), a benchmark for assessing whether frontier AI models can execute economically valuable software engineering work. Unlike existing evaluations that focus on narrow, well-defined tasks, APEX-SWE assesses two novel task types that reflect real-world software engineering work: (1) Integration tasks (n=100), which require constructing end-to-end systems across heterogeneous cloud primitives, business applications, and infrastructure-as-code services, and (2) Observability tasks (n=100), which require debugging production failures using telemetry signals such as logs and dashboards, as well as unstructured context. We evaluated eight frontier models on APEX-SWE. Gemini 3 Pro (Thinking = High) performs best, with a Pass@1 score of 25\%. Our analysis shows that strong performance is primarily driven by epistemic reasoning, defined as the ability to distinguish between assumptions and verified facts, combined with agency to resolve uncertainty prior to acting. We open-source the APEX-SWE evaluation harness and a dev set (n=50).
Paper Structure (58 sections, 3 figures, 9 tables)

This paper contains 58 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Dataset Overview
  3. Integration Tasks Data
  4. Task Environment
  5. Task Sourcing
  6. Authentication & Security
  7. Observability Tasks Data
  8. Task Environment
  9. Task Sourcing
  10. Programming Language
  11. Experimental Setup
  12. Model selection
  13. Leaderboard Evaluation
  14. Rubric Grading
  15. Functional Criteria
  16. ...and 43 more sections

Figures (3)

  • Figure 1: Performance of models on APEX– SWE using Pass@1. Thinking settings are in parentheses.
  • Figure 2: Performance of models on APEX– SWE Observability and APEX– SWE Integration using Pass@1. Thinking settings are in parentheses.
  • Figure 3: Production processes for APEX– SWE Integration and APEX– SWE Observability, leveraging Mercor's platform of experts. The Leaderboard uses Pass@1.