Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CUBE: A Standard for Unifying Agent Benchmarks

Alexandre Lacoste, Nicolas Gontier, Oleh Shliazhko, Aman Jaiswal, Kusha Sareen, Shailesh Nanisetty, Joan Cabezas, Manuel Del Verme, Omar G. Younis, Simone Baratta, Matteo Avalle, Imene Kerboua, Xing Han Lù, Elron Bandel, Michal Shmueli-Scheuer, Asaf Yehudai, Leshem Choshen, Jonathan Lebensold, Sean Hughes, Massimo Caccia, Alexandre Drouin, Siva Reddy, Tao Yu, Yu Su, Graham Neubig, Dawn Song

Abstract

The proliferation of agent benchmarks has created critical fragmentation that threatens research productivity. Each new benchmark requires substantial custom integration, creating an "integration tax" that limits comprehensive evaluation. We propose CUBE (Common Unified Benchmark Environments), a universal protocol standard built on MCP and Gym that allows benchmarks to be wrapped once and used everywhere. By separating task, benchmark, package, and registry concerns into distinct API layers, CUBE enables any compliant platform to access any compliant benchmark for evaluation, RL training, or data generation without custom integration. We call on the community to contribute to the development of this standard before platform-specific implementations deepen fragmentation as benchmark production accelerates through 2026.

CUBE: A Standard for Unifying Agent Benchmarks

Abstract

The proliferation of agent benchmarks has created critical fragmentation that threatens research productivity. Each new benchmark requires substantial custom integration, creating an "integration tax" that limits comprehensive evaluation. We propose CUBE (Common Unified Benchmark Environments), a universal protocol standard built on MCP and Gym that allows benchmarks to be wrapped once and used everywhere. By separating task, benchmark, package, and registry concerns into distinct API layers, CUBE enables any compliant platform to access any compliant benchmark for evaluation, RL training, or data generation without custom integration. We call on the community to contribute to the development of this standard before platform-specific implementations deepen fragmentation as benchmark production accelerates through 2026.
Paper Structure (27 sections, 1 figure, 5 tables)

This paper contains 27 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. The Current Landscape and its Challenges
  3. The CUBE Standard Proposal
  4. Task-Level Interface
  5. Benchmark-Level Interface
  6. Package-Level Standard
  7. CUBE Registry
  8. Python-First Design with RPC Fallback
  9. Adoption Strategy
  10. Call for Collaboration
  11. Initial implementation
  12. Direct Benchmark Creator Outreach
  13. Related Work
  14. Domain-Specific Unification Efforts
  15. Broader Platform Efforts
  16. ...and 12 more sections

Figures (1)

  • Figure 1: Task-level diagram for CUBE's API. Left: Separation between tasks and tools and the possibility to reconfigure the tools. Only the Gym-API is exposed. Right: By implementing base classes, CUBE's implementation can automatically expose an RPC layer, including the well-known MCP API. On the client side, the Gym-API is exposed.