Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interactive Benchmarks

Baoqing Yue, Zihan Zhu, Yifan Zhang, Jichen Feng, Hufei Yang, Mengdi Wang

TL;DR

This work proposes Interactive Benchmarks, a unified evaluation paradigm that assesses model's reasoning ability in an interactive process under budget constraints, and instantiate this framework across two settings: Interactive Proofs, where models interact with a judge to deduce objective truths or answers in logic and mathematics; and Interactive Games, where models reason strategically to maximize long-horizon utilities.

Abstract

Standard benchmarks have become increasingly unreliable due to saturation, subjectivity, and poor generalization. We argue that evaluating model's ability to acquire information actively is important to assess model's intelligence. We propose Interactive Benchmarks, a unified evaluation paradigm that assesses model's reasoning ability in an interactive process under budget constraints. We instantiate this framework across two settings: Interactive Proofs, where models interact with a judge to deduce objective truths or answers in logic and mathematics; and Interactive Games, where models reason strategically to maximize long-horizon utilities. Our results show that interactive benchmarks provide a robust and faithful assessment of model intelligence, revealing that there is still substantial room to improve in interactive scenarios. Project page: https://github.com/interactivebench/interactivebench

Interactive Benchmarks

TL;DR

This work proposes Interactive Benchmarks, a unified evaluation paradigm that assesses model's reasoning ability in an interactive process under budget constraints, and instantiate this framework across two settings: Interactive Proofs, where models interact with a judge to deduce objective truths or answers in logic and mathematics; and Interactive Games, where models reason strategically to maximize long-horizon utilities.

Abstract

Standard benchmarks have become increasingly unreliable due to saturation, subjectivity, and poor generalization. We argue that evaluating model's ability to acquire information actively is important to assess model's intelligence. We propose Interactive Benchmarks, a unified evaluation paradigm that assesses model's reasoning ability in an interactive process under budget constraints. We instantiate this framework across two settings: Interactive Proofs, where models interact with a judge to deduce objective truths or answers in logic and mathematics; and Interactive Games, where models reason strategically to maximize long-horizon utilities. Our results show that interactive benchmarks provide a robust and faithful assessment of model intelligence, revealing that there is still substantial room to improve in interactive scenarios. Project page: https://github.com/interactivebench/interactivebench
Paper Structure (33 sections, 9 equations, 5 figures, 3 tables)

This paper contains 33 sections, 9 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Interactive Benchmarks
  3. Interactive proofs.
  4. Interactive games.
  5. Interactive Proofs: Logic
  6. Interactive Proofs: Math
  7. Interactive Games: Texas Hold'em
  8. Interactive Games: Trust Game
  9. Experiments
  10. Setup
  11. Results
  12. Interactive Proofs: Logic
  13. Interactive Proofs: Math
  14. Interactive Games: Texas Hold'em
  15. Interactive Games: Trust Game
  16. ...and 18 more sections

Figures (5)

  • Figure 1: Overview of the Interactive Benchmarks Framework. Interactive benchmarks acts as a sequential decision process. Left: In Interactive Proofs, the agent queries a Judge to converge on an objective truth (minimizing uncertainty). Right: In Interactive Games, the agent acts in a stochastic or adversarial environment to maximize long-term utility.
  • Figure 2: Evaluation results on Situation Puzzle. We report both success rate and interaction efficiency under a fixed $20$-turn budget with a fixed judge.
  • Figure 3: Interactive vs. pass@$k$ evaluation under the same budget constraint. Left: accuracy under interactive evaluation and pass@$k$, plus the underestimation gap. Right: interaction efficiency measured by average rounds among correct trials.
  • Figure 4: Comparison of six LLM poker agents across 10 independent tables (bars: mean, error bars: standard deviation). (a) Average winnings per hand, (b) VPIP rate, (c) response latency, (d) fold rate. Gemini-3-flash, Grok-4.1-fast, and GPT-5-mini are profitable on average, while GPT-5-mini shows the most aggressive profile (highest VPIP, lowest fold rate).
  • Figure 5: Trust Game tournament results with heuristic baselines and behavioral statistics.