Learnable Behavior Control: Breaking Atari Human World Records via Sample-Efficient Behavior Selection

TL;DR

A general framework called Learnable Behavioral Control (LBC) is proposed to address the limitation and enables a significantly enlarged behavior selection space via formulating a hybrid behavior mapping from all policies; and constructs a unified learnable process for behavior selection.

Abstract

The exploration problem is one of the main challenges in deep reinforcement learning (RL). Recent promising works tried to handle the problem with population-based methods, which collect samples with diverse behaviors derived from a population of different exploratory policies. Adaptive policy selection has been adopted for behavior control. However, the behavior selection space is largely limited by the predefined policy population, which further limits behavior diversity. In this paper, we propose a general framework called Learnable Behavioral Control (LBC) to address the limitation, which a) enables a significantly enlarged behavior selection space via formulating a hybrid behavior mapping from all policies; b) constructs a unified learnable process for behavior selection. We introduce LBC into distributed off-policy actor-critic methods and achieve behavior control via optimizing the selection of the behavior mappings with bandit-based meta-controllers. Our agents have achieved 10077.52% mean human normalized score and surpassed 24 human world records within 1B training frames in the Arcade Learning Environment, which demonstrates our significant state-of-the-art (SOTA) performance without degrading the sample efficiency.

Figures (33)

• Figure 1: Performance on the 57 Atari games. Our method achieves the highest mean human normalized scores agent57, is the first to breakthrough 24 human world records atarihuman, and demands the least training data.
• Figure 2: A General Architecture of Our Algorithm.
• Figure 3: The learning curves in Atari. Curves are smoothed with a moving average over 5 points.
• Figure 4: Behavior entropy and scores curve across training for different games where we achieved unprecedented performance. The names of the axes are the same as that of the leftmost figure.
• Figure 5: Ablation Results. All the results are scaled by the main algorithm to improve readability.

Theorems & Definitions (18)

• Definition 3.1: Behavior Space Construction
• Definition 3.2: Behavior Selection
• Proposition 1: Policy Model Selection
• Proposition 2: Behavior Mapping Optimization
• proof : Proof of Proposition \ref{['proposition: Behavior Control via policy models Selection']}
• Corollary 1: Behavior Circling in Policy Model Selection
• proof : Proof of Proposition \ref{['proposition: Behavior Control via behavior mapping optimization']}
• Corollary 2: Behavior Mapping Optimization Is An Antidote for Behavior Circling
• Proposition 3: Comparison of Behavior Space
• proof : Proof of Proposition \ref{['Proposition: Comparison of Behavior Space']}