Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AGI-Elo: How Far Are We From Mastering A Task?

Shuo Sun, Yimin Zhao, Christina Dao Wen Lee, Jiawei Sun, Chengran Yuan, Zefan Huang, Dongen Li, Justin KW Yeoh, Alok Prakash, Thomas W. Malone, Marcelo H. Ang

TL;DR

AGI-Elo introduces a unified probabilistic rating system that jointly estimates test-case difficulty $R_t$ and agent competency $R_a$ via inter-category matches, using $S=f(M)$ and Gaussian ratings with Glicko-like updates. It predicts agent performance with $\mathbb{E}[M_a] = f^{-1}(\mathbb{E}[S_a])$ where $\mathbb{E}[S_a] = \frac{1}{1+10^{(R_t-R_a)/400}}$, and quantifies long-tail competency gaps to oracle-level mastery. Experiments across six tasks in vision, language, and action reveal long-tail difficulty distributions and nontrivial gaps between current models and task mastery, while reliability analyses show strong rank consistency and predictive accuracy. The framework offers a scalable, interpretable benchmark for tracking progress toward true AGI task mastery and identifying the remaining challenges across diverse domains.

Abstract

As the field progresses toward Artificial General Intelligence (AGI), there is a pressing need for more comprehensive and insightful evaluation frameworks that go beyond aggregate performance metrics. This paper introduces a unified rating system that jointly models the difficulty of individual test cases and the competency of AI models (or humans) across vision, language, and action domains. Unlike existing metrics that focus solely on models, our approach allows for fine-grained, difficulty-aware evaluations through competitive interactions between models and tasks, capturing both the long-tail distribution of real-world challenges and the competency gap between current models and full task mastery. We validate the generalizability and robustness of our system through extensive experiments on multiple established datasets and models across distinct AGI domains. The resulting rating distributions offer novel perspectives and interpretable insights into task difficulty, model progression, and the outstanding challenges that remain on the path to achieving full AGI task mastery.

AGI-Elo: How Far Are We From Mastering A Task?

TL;DR

AGI-Elo introduces a unified probabilistic rating system that jointly estimates test-case difficulty and agent competency via inter-category matches, using and Gaussian ratings with Glicko-like updates. It predicts agent performance with where , and quantifies long-tail competency gaps to oracle-level mastery. Experiments across six tasks in vision, language, and action reveal long-tail difficulty distributions and nontrivial gaps between current models and task mastery, while reliability analyses show strong rank consistency and predictive accuracy. The framework offers a scalable, interpretable benchmark for tracking progress toward true AGI task mastery and identifying the remaining challenges across diverse domains.

Abstract

As the field progresses toward Artificial General Intelligence (AGI), there is a pressing need for more comprehensive and insightful evaluation frameworks that go beyond aggregate performance metrics. This paper introduces a unified rating system that jointly models the difficulty of individual test cases and the competency of AI models (or humans) across vision, language, and action domains. Unlike existing metrics that focus solely on models, our approach allows for fine-grained, difficulty-aware evaluations through competitive interactions between models and tasks, capturing both the long-tail distribution of real-world challenges and the competency gap between current models and full task mastery. We validate the generalizability and robustness of our system through extensive experiments on multiple established datasets and models across distinct AGI domains. The resulting rating distributions offer novel perspectives and interpretable insights into task difficulty, model progression, and the outstanding challenges that remain on the path to achieving full AGI task mastery.
Paper Structure (55 sections, 25 equations, 9 figures, 8 tables)

This paper contains 55 sections, 25 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Existing gaps in AI evaluation
  3. Our contributions
  4. Rating systems explained
  5. Conventional rating systems
  6. Properties & utilities
  7. AGI-Elo rating system design
  8. Test cases vs. agents
  9. Conversion to match results
  10. Rating update
  11. Prediction
  12. Experiments
  13. Experimental setup
  14. Rating distributions
  15. Reliability of the rating system
  16. ...and 40 more sections

Figures (9)

  • Figure 1: In this paper, we address long-standing questions regarding the current capabilities of AGI and humans on challenging tasks by proposing a standardized framework to quantitatively assess task difficulty, evaluate AGI competency, and identify gaps to task mastery.
  • Figure 2: Illustration of the proposed AGI-Elo rating system.
  • Figure 3: Visualization of the estimated test case rating distribution and agent ratings on six distinct datasets. The percentile curve represents the cumulative percentage of test cases up to each rating level. For each agent, the portion of the test cases and the percentile curve that lies to the right represents the fraction of the dataset that remains difficult (below 50% confidence).
  • Figure 4: Evaluation of the reliability as a function of the percentage of completed matches.
  • Figure 5: Visualization of the predicted (theoretical) agent performances based on the differences between agents and test cases vs. the empirical performance obtained on each dataset.
  • ...and 4 more figures