The Formalism-Implementation Gap in Reinforcement Learning Research

TL;DR

The paper identifies a formalism-implementation gap in reinforcement learning, arguing that results tied to a single benchmark like ALE often fail to generalize due to inconsistent mappings between abstract MDP/POMDP formalisms and practical implementations. It advocates explicit mappings between the formalism, benchmarks, and algorithm implementations, plus thorough reporting of hyperparameters, discount factors, and evaluation protocols. By analyzing ALE-specific design choices (frame skip, frame stacking, reward clipping, life-loss termination, and action sets), the authors show how these factors influence observed performance and learning dynamics. The work promotes insight-oriented evaluation, per-game analysis, and extendable benchmarks to improve transferability and scientific understanding, ultimately aiming for RL methods that generalize to real-world problems beyond flame-graphs of performance on a saturated benchmark.

Abstract

The last decade has seen an upswing in interest and adoption of reinforcement learning (RL) techniques, in large part due to its demonstrated capabilities at performing certain tasks at "super-human levels". This has incentivized the community to prioritize research that demonstrates RL agent performance, often at the expense of research aimed at understanding their learning dynamics. Performance-focused research runs the risk of overfitting on academic benchmarks -- thereby rendering them less useful -- which can make it difficult to transfer proposed techniques to novel problems. Further, it implicitly diminishes work that does not push the performance-frontier, but aims at improving our understanding of these techniques. This paper argues two points: (i) RL research should stop focusing solely on demonstrating agent capabilities, and focus more on advancing the science and understanding of reinforcement learning; and (ii) we need to be more precise on how our benchmarks map to the underlying mathematical formalisms. We use the popular Arcade Learning Environment (ALE; Bellemare et al., 2013) as an example of a benchmark that, despite being increasingly considered "saturated", can be effectively used for developing this understanding, and facilitating the deployment of RL techniques in impactful real-world problems.

Figures (4)

• Figure 1: RL Algorithms are typically defined using the mathematical formalism of MDPs, but implemented (somewhat) independently with code, and evaluated on standard benchmarks such as the ALE. If we want to maximize the transferability of our algorithms, we need to be explicit about the MDP-benchmark and algorithm-implementation mappings.
• Figure 2: The impact of frame skip and frame stacking on DQN. Evaluated on Dopamine castro2018dopamine on the Atari-5 games aitchison2023atari5; solid lines are mean performance over 5 independent seeds, shaded areas represent 95% confidence intervals.
• Figure 3: Varying the definition of $\mathcal{A}$ and $\mathcal{P}$ can have a large impact on reported performance. Evaluated on Dopamine castro2018dopamine with DQN, solid lines are mean performance over 5 independent seeds, shaded areas represent 95% confidence intervals.
• Figure 4: Evaluating the comparison of DQN and Rainbow hessel2018rainbow over different subsets of games. In each row, the leftmost column depicts the human-normalized Interquartile mean with 95% stratified bootstrap confidence intervals agarwal2021deep, computed over the games on the same row, with 5 independent seeds for each. The ranking of the algorithms can dramatically change depending on the subset of games chosen.