Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Hierarchy of Agentic Capabilities: Evaluating Frontier Models on Realistic RL Environments

Logan Ritchie, Sushant Mehta, Nick Heiner, Mason Yu, Edwin Chen

TL;DR

This work evaluates frontier AI agents in a realistic Corecraft RL environment with $150$ workplace tasks, identifying a five-level hierarchy of agentic capabilities and a persistent $40\%$ failure rate even for top models. It introduces a task-centric environment design and a modular tool interface (MCP) to expose concrete failure modes across tool use, planning, adaptability, groundedness, and common-sense reasoning. Failures cluster predictably by capability level, with lower levels constraining weaker models and higher-level gaps (notably common-sense reasoning) limiting stronger models, guiding targeted improvement and curriculum design. The study links capability diagnostics to production deployment patterns, arguing for bounded autonomy and structured human oversight as a practical, near-term path toward reliable, real-world AI agents in workplaces.

Abstract

The advancement of large language model (LLM) based agents has shifted AI evaluation from single-turn response assessment to multi-step task completion in interactive environments. We present an empirical study evaluating frontier AI models on 150 workplace tasks within a realistic e-commerce RL environment from Surge. Our analysis reveals an empirically-derived \emph{hierarchy of agentic capabilities} that models must master for real-world deployment: (1) tool use, (2) planning and goal formation, (3) adaptability, (4) groundedness, and (5) common-sense reasoning. Even the best-performing models fail approximately 40\% of the tasks, with failures clustering predictably along this hierarchy. Weaker models struggle with fundamental tool use and planning, whereas stronger models primarily fail on tasks requiring contextual inference beyond explicit instructions. We introduce a task-centric design methodology for RL environments that emphasizes diversity and domain expert contributions, provide detailed failure analysis, and discuss implications for agent development. Our findings suggest that while current frontier models can demonstrate coherent multi-step behavior, substantial capability gaps remain before achieving human-level task completion in realistic workplace settings.

The Hierarchy of Agentic Capabilities: Evaluating Frontier Models on Realistic RL Environments

TL;DR

This work evaluates frontier AI agents in a realistic Corecraft RL environment with workplace tasks, identifying a five-level hierarchy of agentic capabilities and a persistent failure rate even for top models. It introduces a task-centric environment design and a modular tool interface (MCP) to expose concrete failure modes across tool use, planning, adaptability, groundedness, and common-sense reasoning. Failures cluster predictably by capability level, with lower levels constraining weaker models and higher-level gaps (notably common-sense reasoning) limiting stronger models, guiding targeted improvement and curriculum design. The study links capability diagnostics to production deployment patterns, arguing for bounded autonomy and structured human oversight as a practical, near-term path toward reliable, real-world AI agents in workplaces.

Abstract

The advancement of large language model (LLM) based agents has shifted AI evaluation from single-turn response assessment to multi-step task completion in interactive environments. We present an empirical study evaluating frontier AI models on 150 workplace tasks within a realistic e-commerce RL environment from Surge. Our analysis reveals an empirically-derived \emph{hierarchy of agentic capabilities} that models must master for real-world deployment: (1) tool use, (2) planning and goal formation, (3) adaptability, (4) groundedness, and (5) common-sense reasoning. Even the best-performing models fail approximately 40\% of the tasks, with failures clustering predictably along this hierarchy. Weaker models struggle with fundamental tool use and planning, whereas stronger models primarily fail on tasks requiring contextual inference beyond explicit instructions. We introduce a task-centric design methodology for RL environments that emphasizes diversity and domain expert contributions, provide detailed failure analysis, and discuss implications for agent development. Our findings suggest that while current frontier models can demonstrate coherent multi-step behavior, substantial capability gaps remain before achieving human-level task completion in realistic workplace settings.
Paper Structure (32 sections, 2 figures)

This paper contains 32 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Agent Benchmarks and Evaluation
  4. Production Agent Deployment
  5. Environment Design
  6. Design Philosophy
  7. Environment Architecture
  8. The Corecraft Environment
  9. Tool Interface
  10. Task Design
  11. Experimental Setup
  12. Models Evaluated
  13. Evaluation Protocol
  14. Evaluation Criteria
  15. Results
  16. ...and 17 more sections

Figures (2)

  • Figure 1: Task completion rates across frontier models on 150 workplace tasks (December 2025 Update). GPT-5.2 and Claude Opus 4.5 maintain the lead, while mid-tier models like Gemini 3 Pro and Nova 2 Pro demonstrate significant progress over previous generations.
  • Figure 2: The hierarchy of agentic capabilities. Weaker models fail at basic tool use and planning; stronger models reach common-sense reasoning as their limiting factor.