Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Well Does Agent Development Reflect Real-World Work?

Zora Zhiruo Wang, Sanidhya Vijayvargiya, Aspen Chen, Hanmo Zhang, Venu Arvind Arangarajan, Jett Chen, Valerie Chen, Diyi Yang, Daniel Fried, Graham Neubig

TL;DR

This work systematically study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills, revealing substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated.

Abstract

AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor market as a whole. In this work, we systematically study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills. We first analyze 43 benchmarks and 72,342 tasks, measuring their alignment with human employment and capital allocation across all 1,016 real-world occupations in the U.S. labor market. We reveal substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated. Within work areas that agents currently target, we further characterize current agent utility by measuring their autonomy levels, providing practical guidance for agent interaction strategies across work scenarios. Building on these findings, we propose three measurable principles for designing benchmarks that better capture socially important and technically challenging forms of work: coverage, realism, and granular evaluation.

How Well Does Agent Development Reflect Real-World Work?

TL;DR

This work systematically study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills, revealing substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated.

Abstract

AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor market as a whole. In this work, we systematically study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills. We first analyze 43 benchmarks and 72,342 tasks, measuring their alignment with human employment and capital allocation across all 1,016 real-world occupations in the U.S. labor market. We reveal substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated. Within work areas that agents currently target, we further characterize current agent utility by measuring their autonomy levels, providing practical guidance for agent interaction strategies across work scenarios. Building on these findings, we propose three measurable principles for designing benchmarks that better capture socially important and technically challenging forms of work: coverage, realism, and granular evaluation.
Paper Structure (26 sections, 2 equations, 7 figures, 4 tables)

This paper contains 26 sections, 2 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Bridging Agents to Real-World Work
  3. The Landscape of Human Work
  4. Agent Development Effort for Work
  5. Skewed Emphasis in Agent Development
  6. Benchmarks Focus on Software Engineering but Miss Other Digitized Domains
  7. Benchmarks Overfocus on a Narrow Set of Skills
  8. From Work Relevance to Representativeness
  9. How Autonomous Can Agents Be?
  10. Grounding Agent Autonomy in Task Complexity
  11. Calibrating Agent Autonomy for Real-World Tasks
  12. Discussion
  13. Related Work
  14. Principles to Benchmark Agents for Work
  15. Conclusion
  16. ...and 11 more sections

Figures (7)

  • Figure 1: Mapping agent benchmarks to work domains and skills in O*NET.
  • Figure 2: Agent benchmarking effort is concentrated in mathematical and engineering domains, despite broad human employment and economic value across other domains.
  • Figure 3: Agent benchmarking effort emphasizes two granular skills (getting information and working with computers), which together, only cover $<5$% human employment, creating a heavily imbalanced distribution across skills. See §\ref{['app:2.1:onet-skills']} for full skill names.
  • Figure 4: Number of domains (top) and skills (bottom) an average benchmark example map to.
  • Figure 5: Exemplar workflow induced from agent low-level action trajectory.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Definition 1
  • Definition 2