Agent57: Outperforming the Atari Human Benchmark

TL;DR

Agent57 addresses the Atari 57 benchmark by extending NGU with a state-action value decomposition, an adaptive meta-controller, and longer backpropagation windows to tackle long-term credit assignment and exploration. The core contributions are a two-network intrinsic/extrinsic Q parameterization, a per-actor policy-selection bandit, and empirical evidence that a longer backprop window improves stability and learning. Agent57 achieves 100% capped human-normalized score and outperforms prior baselines across the tail of games, demonstrating strong generalization. The work highlights the value of adaptive, multi-policy training for broad-spectrum Atari competency and points to future gains in data efficiency and planning-based extensions.

Abstract

Atari games have been a long-standing benchmark in the reinforcement learning (RL) community for the past decade. This benchmark was proposed to test general competency of RL algorithms. Previous work has achieved good average performance by doing outstandingly well on many games of the set, but very poorly in several of the most challenging games. We propose Agent57, the first deep RL agent that outperforms the standard human benchmark on all 57 Atari games. To achieve this result, we train a neural network which parameterizes a family of policies ranging from very exploratory to purely exploitative. We propose an adaptive mechanism to choose which policy to prioritize throughout the training process. Additionally, we utilize a novel parameterization of the architecture that allows for more consistent and stable learning.

Figures (14)

• Figure 1: Number of games where algorithms are better than the human benchmark throughout training for Agent57 and state-of-the-art baselines on the 57 Atari games.
• Figure 2: A schematic depiction of a distributed deep RL agent.
• Figure 3: Capped human normalized score where we observe at which point the agent surpasses the human benchmark on the last 6 games.
• Figure 4: Performance progression on the 10-game challenging set obtained from incorporating each one of the improvements.
• Figure 5: Extrinsic returns for the exploitative ($\beta_0=0$) and most exploratory ($\beta_{31}=\beta$) on "random coin" for different values of the intrinsic reward weight, $\beta$. (Top) NGU(Bottom) NGU with Separate networks for intrinsic and extrinsic values.