Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Active Evaluation of General Agents: Problem Definition and Comparison of Baseline Algorithms

Marc Lanctot, Kate Larson, Ian Gemp, Michael Kaisers

TL;DR

The paper tackles the challenge of efficiently evaluating general agents across multi-task benchmarks by formalizing an online active-evaluation framework where the evaluator selects tasks and agent pairs to sample and continually updates a ranking. It introduces ground-truth-based performance measures using $K_d$, $K_n$, and theAverage Generalized Ranking Error (AGRE) to quantify ranking quality, and develops synthetic data models (Mallows and Plackett-Luce) to test algorithms. By extending offline voting and game-theoretic evaluation methods (VasE, Nash Averaging, Metritocracy) to online settings, it compares baselines (Elo variants, UCB) and online growing-batch algorithms (SCO, Maximal Lotteries, Nash Averaging) on synthetic data and Atari Agent57, revealing that BatchElo is consistently reliable, SCO variants often outperform Elo on real data, and ProportionalRepresentation shines under high task variation. The findings guide practical choices for efficient, robust multi-task evaluation and highlight directions for theoretical guarantees, adaptive task selection, and clone-resilience enhancements.

Abstract

As intelligent agents become more generally-capable, i.e. able to master a wide variety of tasks, the complexity and cost of properly evaluating them rises significantly. Tasks that assess specific capabilities of the agents can be correlated and stochastic, requiring many samples for accurate comparisons, leading to added costs. In this paper, we propose a formal definition and a conceptual framework for active evaluation of agents across multiple tasks, which assesses the performance of ranking algorithms as a function of number of evaluation data samples. Rather than curating, filtering, or compressing existing data sets as a preprocessing step, we propose an online framing: on every iteration, the ranking algorithm chooses the task and agents to sample scores from. Then, evaluation algorithms report a ranking of agents on each iteration and their performance is assessed with respect to the ground truth ranking over time. Several baselines are compared under different experimental contexts, with synthetic generated data and simulated online access to real evaluation data from Atari game-playing agents. We find that the classical Elo rating system -- while it suffers from well-known failure modes, in theory -- is a consistently reliable choice for efficient reduction of ranking error in practice. A recently-proposed method, Soft Condorcet Optimization, shows comparable performance to Elo on synthetic data and significantly outperforms Elo on real Atari agent evaluation. When task variation from the ground truth is high, selecting tasks based on proportional representation leads to higher rate of ranking error reduction.

Active Evaluation of General Agents: Problem Definition and Comparison of Baseline Algorithms

TL;DR

The paper tackles the challenge of efficiently evaluating general agents across multi-task benchmarks by formalizing an online active-evaluation framework where the evaluator selects tasks and agent pairs to sample and continually updates a ranking. It introduces ground-truth-based performance measures using , , and theAverage Generalized Ranking Error (AGRE) to quantify ranking quality, and develops synthetic data models (Mallows and Plackett-Luce) to test algorithms. By extending offline voting and game-theoretic evaluation methods (VasE, Nash Averaging, Metritocracy) to online settings, it compares baselines (Elo variants, UCB) and online growing-batch algorithms (SCO, Maximal Lotteries, Nash Averaging) on synthetic data and Atari Agent57, revealing that BatchElo is consistently reliable, SCO variants often outperform Elo on real data, and ProportionalRepresentation shines under high task variation. The findings guide practical choices for efficient, robust multi-task evaluation and highlight directions for theoretical guarantees, adaptive task selection, and clone-resilience enhancements.

Abstract

As intelligent agents become more generally-capable, i.e. able to master a wide variety of tasks, the complexity and cost of properly evaluating them rises significantly. Tasks that assess specific capabilities of the agents can be correlated and stochastic, requiring many samples for accurate comparisons, leading to added costs. In this paper, we propose a formal definition and a conceptual framework for active evaluation of agents across multiple tasks, which assesses the performance of ranking algorithms as a function of number of evaluation data samples. Rather than curating, filtering, or compressing existing data sets as a preprocessing step, we propose an online framing: on every iteration, the ranking algorithm chooses the task and agents to sample scores from. Then, evaluation algorithms report a ranking of agents on each iteration and their performance is assessed with respect to the ground truth ranking over time. Several baselines are compared under different experimental contexts, with synthetic generated data and simulated online access to real evaluation data from Atari game-playing agents. We find that the classical Elo rating system -- while it suffers from well-known failure modes, in theory -- is a consistently reliable choice for efficient reduction of ranking error in practice. A recently-proposed method, Soft Condorcet Optimization, shows comparable performance to Elo on synthetic data and significantly outperforms Elo on real Atari agent evaluation. When task variation from the ground truth is high, selecting tasks based on proportional representation leads to higher rate of ranking error reduction.
Paper Structure (35 sections, 19 equations, 8 figures, 5 tables, 1 algorithm)

This paper contains 35 sections, 19 equations, 8 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Terminology
  3. Evaluation Systems for General Agents
  4. Active Learning
  5. Multi-armed Bandits
  6. Active Evaluation
  7. Metrics
  8. Models for Synthetic Data
  9. Mallows
  10. Plackett-Luce
  11. Algorithms for Active Evaluation
  12. Baselines and Bandit Approaches
  13. Stochastic Bandit-based Approaches
  14. Dueling Bandit Approaches
  15. Growing Batch and Online Algorithms
  16. ...and 20 more sections

Figures (8)

  • Figure 1: Example of synthetic data generation process with $m = 4$ and $n = 5$. The task rankings $( \succ^*_{v_1}, \succ^*_{v_2}, \cdots, \succ^*_{v_5})$ have Kendall-tau distances 4, 1, 4, 1, and 5, respectively, from $\succ^*$.
  • Figure 2: Exp Condition 1: Generalized top-$k$ ranking error (GRE) over iterations for several algorithms. Each line is a mean over the same 100 seeds with 95% confidence intervals in shaded area. Each data point is a sliding-window average of the most recent 250 values of GRE for the reported ranking $\succ^t$ at iteration $t$. The top two graphs use dispersion $\phi=0.3$, with $k \in \{3, 8\}$, where $k=8$ corresponds to normalized Kendall-tau distance. The bottom two use $\phi=0.6$ and vary $k$ similarly.
  • Figure 3: Exp Condition 2: Generalized top-$k$ ranking error (GRE) over iterations for several algorithms. Each data point is a sliding-window average of the most recent 250 values of GRE for the reported ranking $\succ^t$ at iteration $t$. Both graphs use $k=3$. Please note the log scale for both axes.
  • Figure 4: Task variation for samples from a Mallows model with $m = 8, n = 50$. Each graph shows a histogram of the 50 Kendall-tau distances between the task ranking and the ground truth ranking, $K_d(\succ^*, \succ^*_v)$.
  • Figure 5: Task variation for Atari Agent 57 data generator with $m = 8, n = 57$.
  • ...and 3 more figures

Theorems & Definitions (5)

  • definition 1
  • definition 2
  • definition 3: Top-$k$ Identification Error
  • definition 4: Generalized Top-$k$ Ranking Error
  • definition 5: Average Generalized Top-$k$ Ranking Error