On the Interplay Between Sparsity and Training in Deep Reinforcement Learning

TL;DR

The paper investigates how sparse connectivity interacts with training strategy in deep reinforcement learning for image-based domains. It compares several fixed-sparsity baselines (Random, Spatial, Predictive) and end-to-end sparsity (L1-regularization) within a DQN framework across MinAtar Breakout and Space-Invaders, considering both fixed and learned hidden weights. The key finding is that sparse structure significantly affects learning performance and that the optimal topology depends on whether the hidden weights are fixed or learned, with spatial sparsity not universally superior and end-to-end sparsity (L1) capable of matching dense performance when learned. These results guide design choices for sparse RL agents and motivate further exploration of end-to-end sparse training and representation-aligned sparsity strategies.

Abstract

We study the benefits of different sparse architectures for deep reinforcement learning. In particular, we focus on image-based domains where spatially-biased and fully-connected architectures are common. Using these and several other architectures of equal capacity, we show that sparse structure has a significant effect on learning performance. We also observe that choosing the best sparse architecture for a given domain depends on whether the hidden layer weights are fixed or learned.

Figures (11)

• Figure 1: Average return for DQN architectures whose hidden layer is randomly initialized and frozen in each environment: Breakout (left) and Space-Invaders (right).
• Figure 2: Average return corresponding to DQN architectures whose hidden layer is learned end-to-end in each environment: Breakout (left) and Space-Invaders (right). Horizontal dashed lines indicate the final performances when the hidden layer weights are never learned.
• Figure 3: Visualization of the Breakout and Space-Invaders environments minatar.
• Figure 4: One of the predictive masks in Breakout. The inputs in yellow are those that are "on" in the mask. Prediction Adapted Neighborhoods found this subset to help predict the next values of the entry marked by the red "X".
• Figure 5: One of the spatial neighborhoods in Breakout. The inputs that are "on" are shown in yellow---these are located closest to entries marked by the red "X".