Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning

TL;DR

This work tackles the challenge of scalable deep reinforcement learning by disentangling the roles of architectural design and dynamic sparse training (DST). It shows that architecture largely governs encoder self-supervised representations and overall scalability, while DST provides module-specific benefits—most notably improving TD-based critic learning but offering limited gains for well-designed actors. The authors propose Module-Specific Training (MST), which assigns dense training to encoders, DST to critics, and static sparse training to actors, achieving substantial scalability on MR.Q and generalizing to SAC and DDPG without modifying RL algorithms. The findings yield practical guidance: combine strong architectural foundations with module-tailored training to unlock scalable DRL, including scaling to hundreds of millions of parameters. This work thus bridges the gap between architectural advances and dynamic training, highlighting how RL-specific DL mechanisms can push DRL toward real-world, large-scale deployment.

Abstract

Scaling neural networks has driven breakthrough advances in machine learning, yet this paradigm fails in deep reinforcement learning (DRL), where larger models often degrade performance due to unique optimization pathologies such as plasticity loss. While recent works show that dynamically adapting network topology during training can mitigate these issues, existing studies have three critical limitations: (1) applying uniform dynamic training strategies across all modules despite encoder, critic, and actor following distinct learning paradigms, (2) focusing evaluation on basic architectures without clarifying the relative importance and interaction between dynamic training and architectural improvements, and (3) lacking systematic comparison between different dynamic approaches including sparse-to-sparse, dense-to-sparse, and sparse-to-dense. Through comprehensive investigation across modules and architectures, we reveal that dynamic sparse training strategies provide module-specific benefits that complement the primary scalability foundation established by architectural improvements. We finally distill these insights into Module-Specific Training (MST), a practical framework that further exploits the benefits of architectural improvements and demonstrates substantial scalability gains across diverse RL algorithms without algorithmic modifications.

Figures (26)

• Figure 1: Network architecture variants for our investigation, incorporating three key architectural improvements: activation functions, layer normalization, and residual connections.
• Figure 2: Impact of encoder architecture types and training regimes on agent performance at default size. The results demonstrate that architectural improvements (progression from Type 1 to Type 6) yield substantially greater performance gains than differences between training regimes.
• Figure 3: Representational collapse causes ineffective training without layer normalization.
• Figure 4: Scaling behavior of agent performance as encoder parameter count increases across three orders of magnitude. While introducing network sparsity and dynamicity can sometimes outperform dense training, architectural design remains the fundamental determinant of scalability potential.
• Figure 5: Comparison of critic architecture types and training regimes at default network size. With a strong encoder providing effective state and action representations, critic networks reliably achieve stable optimization. Beyond layer normalization, other architectural advances and training regime variations show limited impact on critic performance at this default scale.