Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

General Agent Evaluation

Elron Bandel, Asaf Yehudai, Lilach Eden, Yehoshua Sagron, Yotam Perlitz, Elad Venezian, Natalia Razinkov, Natan Ergas, Shlomit Shachor Ifergan, Segev Shlomov, Michal Jacovi, Leshem Choshen, Liat Ein-Dor, Yoav Katz, Michal Shmueli-Scheuer

TL;DR

This paper proposes conceptual principles for general-agent evaluation, a Unified Protocol enabling agent-benchmark integration, and Exgentic - a practical framework for general agent evaluation, and releases the evaluation protocol, framework, and leaderboard to establish a foundation for systematic research on general-purpose agents.

Abstract

The promise of general-purpose agents - systems that perform tasks in unfamiliar environments without domain-specific engineering - remains largely unrealized. Existing agents are predominantly specialized, and while emerging implementations like OpenAI SDK Agent and Claude Code hint at broader capabilities, no systematic evaluation of their general performance has been pursued. Current agentic benchmarks assume domain-specific integration, encoding task information in ways that preclude fair evaluation of general agents. This paper frames general-agent evaluation as a first-class research objective. We propose conceptual principles for such evaluation, a Unified Protocol enabling agent-benchmark integration, and Exgentic - a practical framework for general agent evaluation. We benchmark five prominent agent implementations across six environments as the first Open General Agent Leaderboard. Our experiments show that general agents generalize across diverse environments, achieving performance comparable to domain-specific agents without any environment-specific tuning. We release our evaluation protocol, framework, and leaderboard to establish a foundation for systematic research on general-purpose agents.

General Agent Evaluation

TL;DR

This paper proposes conceptual principles for general-agent evaluation, a Unified Protocol enabling agent-benchmark integration, and Exgentic - a practical framework for general agent evaluation, and releases the evaluation protocol, framework, and leaderboard to establish a foundation for systematic research on general-purpose agents.

Abstract

The promise of general-purpose agents - systems that perform tasks in unfamiliar environments without domain-specific engineering - remains largely unrealized. Existing agents are predominantly specialized, and while emerging implementations like OpenAI SDK Agent and Claude Code hint at broader capabilities, no systematic evaluation of their general performance has been pursued. Current agentic benchmarks assume domain-specific integration, encoding task information in ways that preclude fair evaluation of general agents. This paper frames general-agent evaluation as a first-class research objective. We propose conceptual principles for such evaluation, a Unified Protocol enabling agent-benchmark integration, and Exgentic - a practical framework for general agent evaluation. We benchmark five prominent agent implementations across six environments as the first Open General Agent Leaderboard. Our experiments show that general agents generalize across diverse environments, achieving performance comparable to domain-specific agents without any environment-specific tuning. We release our evaluation protocol, framework, and leaderboard to establish a foundation for systematic research on general-purpose agents.
Paper Structure (36 sections, 2 equations, 4 figures, 7 tables)

This paper contains 36 sections, 2 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Unified Protocol Methodology
  3. Agent Benchmark Unified Protocol
  4. Methodology for Adapting Existing Benchmarks
  5. Methodology for Adapting Existing Agents
  6. Exgentic Framework
  7. Solving the Integration Problem
  8. Experimental Setup
  9. Benchmarks
  10. Agents
  11. Agent Components
  12. Metrics
  13. Results
  14. Key Leaderboard Findings
  15. No Single Agent Dominates Across Task Domains
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Cost-performance tradeoffs across agent-model configurations. The Pareto frontier (red dashed line) shows optimal tradeoffs: GPT 5.2 configurations offer the best cost-efficiency while Claude Opus 4.5 achieve the highest performance at 3-33$\times$ higher cost.
  • Figure 2: Evolution of Agentic Evaluation. (A) Collection of separate benchmarks, each requiring a custom agent or an agent with specific adaptation per benchmark (HAL) (B) Multiple benchmarks consolidated through a single protocol, such as CLI, or Web (C) Multiple benchmarks consolidated through a common protocol that can be adapted to any agent's protocol (Exgentic).
  • Figure 3: Open General Agent Leaderboard is the first benchmark to consistently test general-agent architectures across key skills in diverse environments.
  • Figure 4: Exgentic architecture. Exgentic defines a unified protocol between agents and benchmarks. The Exgentic Orchestrator connects the agent and the benchmark, first passing the task definition and then mediates the observations and actions that are passed between the benchmark and the agent. Exgentic provides adaptors that convert the Unified Protocol into the specific protocols required by the agents and benchmarks. Finally, the benchmark provides the quality result metrics while the agent provides the agent runtime cost.