Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can we hop in general? A discussion of benchmark selection and design using the Hopper environment

Claas A Voelcker, Marcel Hussing, Eric Eaton

TL;DR

The paper addresses the problem of benchmark representativeness in reinforcement learning by showing that different instantiations of a Hopper benchmark yield divergent algorithm evaluations, threatening the validity of progress claims. It analyzes two Hopper variants (Gym and DM Control), tests four diverse algorithms, and demonstrates that both absolute performance and algorithm rankings can vary substantially across benchmarks. The authors argue for treating benchmarks as a scientific discipline, proposing a common language (purpose, problems, properties, measurable quantities) and emphasizing measurable, cross-benchmark criteria to assess representativeness and guide benchmark design. This work highlights the practical impact of benchmark choices on interpreting progress toward general intelligence and calls for structured, community-wide dialogue to improve RL evaluation practices.

Abstract

Empirical, benchmark-driven testing is a fundamental paradigm in the current RL community. While using off-the-shelf benchmarks in reinforcement learning (RL) research is a common practice, this choice is rarely discussed. Benchmark choices are often done based on intuitive ideas like "legged robots" or "visual observations". In this paper, we argue that benchmarking in RL needs to be treated as a scientific discipline itself. To illustrate our point, we present a case study on different variants of the Hopper environment to show that the selection of standard benchmarking suites can drastically change how we judge performance of algorithms. The field does not have a cohesive notion of what the different Hopper environments are representative - they do not even seem to be representative of each other. Our experimental results suggests a larger issue in the deep RL literature: benchmark choices are neither commonly justified, nor does there exist a language that could be used to justify the selection of certain environments. This paper concludes with a discussion of the requirements for proper discussion and evaluations of benchmarks and recommends steps to start a dialogue towards this goal.

Can we hop in general? A discussion of benchmark selection and design using the Hopper environment

TL;DR

The paper addresses the problem of benchmark representativeness in reinforcement learning by showing that different instantiations of a Hopper benchmark yield divergent algorithm evaluations, threatening the validity of progress claims. It analyzes two Hopper variants (Gym and DM Control), tests four diverse algorithms, and demonstrates that both absolute performance and algorithm rankings can vary substantially across benchmarks. The authors argue for treating benchmarks as a scientific discipline, proposing a common language (purpose, problems, properties, measurable quantities) and emphasizing measurable, cross-benchmark criteria to assess representativeness and guide benchmark design. This work highlights the practical impact of benchmark choices on interpreting progress toward general intelligence and calls for structured, community-wide dialogue to improve RL evaluation practices.

Abstract

Empirical, benchmark-driven testing is a fundamental paradigm in the current RL community. While using off-the-shelf benchmarks in reinforcement learning (RL) research is a common practice, this choice is rarely discussed. Benchmark choices are often done based on intuitive ideas like "legged robots" or "visual observations". In this paper, we argue that benchmarking in RL needs to be treated as a scientific discipline itself. To illustrate our point, we present a case study on different variants of the Hopper environment to show that the selection of standard benchmarking suites can drastically change how we judge performance of algorithms. The field does not have a cohesive notion of what the different Hopper environments are representative - they do not even seem to be representative of each other. Our experimental results suggests a larger issue in the deep RL literature: benchmark choices are neither commonly justified, nor does there exist a language that could be used to justify the selection of certain environments. This paper concludes with a discussion of the requirements for proper discussion and evaluations of benchmarks and recommends steps to start a dialogue towards this goal.

Paper Structure

This paper contains 14 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Empirical case study: The Hopper environment
  4. Comparing the benchmark specifications
  5. Reward-based reinforcement learning
  6. Reward-free reinforcement learning
  7. How do we describe and evaluate benchmarks?
  8. Evaluating the purpose of reinforcement learning
  9. Towards effective benchmarks: Developing a common language
  10. A peek at other fields of science
  11. Conclusion
  12. Implementation details of algorithms
  13. DIAYN sequences
  14. Additional related work on benchmarks

Figures (3)

  • Figure 1: Performance evaluation on the Hopper environment variations. Shaded area denotes a bootstrapped 95% confidence interval of the mean at 95 across 30 seeds with 5000 resamples.
  • Figure 2: Final states for 8 out of 20 "skills" learnt by the DIAYN algorithm on DMC Hopper. In all cases the Hopper immediately moves towards the final configuration without displaying the dynamic skills reported in the original paper.
  • Figure 3: Full visualization of 8 out of 20 skills from one run of DIAYN. Each row is one sequence from the execution of a skill, with each column one frame every 100 environment time steps. For almost all skills we observe that the Hopper maintains a static pose after the first 50-100 time-steps, which is a very different behavior from the one reported in the original paper. The poses are mostly distinct, which does optimize the discriminative objective of the algorithm.

Theorems & Definitions (1)

  • Definition 2.1: Benchmark