Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The AI Productivity Index (APEX)

Bertie Vidgen, Abby Fennelly, Evan Pinnix, Julien Benchek, Daniyal Khan, Zach Richards, Austin Bridges, Calix Huang, Kanishka Sahu, Abhishek Kottamasu, Bo Ma, Ben Hunsberger, Isaac Robinson, Akul Datta, Chirag Mahapatra, Dominic Barton, Cass R. Sunstein, Eric Topol, Brendan Foody, Osvald Nitski

TL;DR

APEX-v1-extended extends a real-world AI productivity benchmark to four professional domains, expanding the heldout set to 400 cases and adding a 100-case devset to support open research. The evaluation combines rubric-based prompts, curated sources, and eight-run LM judging to produce robust, CI-backed scores across 10 frontier models, with GPT-5 (Thinking=High) leading at ~67%. Results reveal substantial gaps between frontier models and routine professional productivity, varying by task difficulty and domain, and show only modest consistency with the earlier devset. The work provides open data and tooling to accelerate research on AI-assisted professional work and highlights the need for further advances before frontier models reliably replace or augment human professionals.

Abstract

We present an extended version of the AI Productivity Index (APEX-v1-extended), a benchmark for assessing whether frontier models are capable of performing economically valuable tasks in four jobs: investment banking associate, management consultant, big law associate, and primary care physician (MD). This technical report details the extensions to APEX-v1, including an increase in the held-out evaluation set from n = 50 to n = 100 cases per job (n = 400 total) and updates to the grading methodology. We present a new leaderboard, where GPT5 (Thinking = High) remains the top performing model with a score of 67.0%. APEX-v1-extended shows that frontier models still have substantial limitations when performing typical professional tasks. To support further research, we are open sourcing n = 25 non-benchmark example cases per role (n = 100 total) along with our evaluation harness.

The AI Productivity Index (APEX)

TL;DR

APEX-v1-extended extends a real-world AI productivity benchmark to four professional domains, expanding the heldout set to 400 cases and adding a 100-case devset to support open research. The evaluation combines rubric-based prompts, curated sources, and eight-run LM judging to produce robust, CI-backed scores across 10 frontier models, with GPT-5 (Thinking=High) leading at ~67%. Results reveal substantial gaps between frontier models and routine professional productivity, varying by task difficulty and domain, and show only modest consistency with the earlier devset. The work provides open data and tooling to accelerate research on AI-assisted professional work and highlights the need for further advances before frontier models reliably replace or augment human professionals.

Abstract

We present an extended version of the AI Productivity Index (APEX-v1-extended), a benchmark for assessing whether frontier models are capable of performing economically valuable tasks in four jobs: investment banking associate, management consultant, big law associate, and primary care physician (MD). This technical report details the extensions to APEX-v1, including an increase in the held-out evaluation set from n = 50 to n = 100 cases per job (n = 400 total) and updates to the grading methodology. We present a new leaderboard, where GPT5 (Thinking = High) remains the top performing model with a score of 67.0%. APEX-v1-extended shows that frontier models still have substantial limitations when performing typical professional tasks. To support further research, we are open sourcing n = 25 non-benchmark example cases per role (n = 100 total) along with our evaluation harness.

Paper Structure

This paper contains 16 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Dataset overview
  3. Selection of experts
  4. Quality control
  5. Selection of sources
  6. Rubric creation
  7. Leaderboard setup
  8. Model selection
  9. Grading responses with an LM judge
  10. Results
  11. Leaderboard results
  12. Performance given task difficulty
  13. Comparison of APEX-v1-extended and APEX-v1-devset
  14. Acknowledgments
  15. LM judge
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Models' mean score on APEX-v1-extended. The labels in parentheses show the "Thinking" setting.
  • Figure 2: Production process for creating APEX-v1-extended.
  • Figure 3: Example prompt and rubric from the Investment banking associate job in APEX-v1-devset, with $10$ criteria. The prompt is passed to models with one source, a financial model in an XLSX file.