Decorrelated Soft Actor-Critic for Efficient Deep Reinforcement Learning

TL;DR

The paper tackles sample inefficiency in deep RL caused by highly correlated representations by introducing online network-wide decorrelation via decorrelated backpropagation (DBP) integrated with soft actor-critic (SAC), forming DSAC. It adds a learnable decorrelation matrix $\mathbf{R}$ to every layer input, updating $\mathbf{R}$ with $\mathbf{R} \leftarrow \mathbf{R} - \eta \mathbf{C}\mathbf{R}$ and minimizing a total decorrelation loss $D = \sum_l d_l$ alongside the RL losses. Empirically, DSAC applied to discrete SAC on Atari 100k achieves faster wall-clock training in five of seven games and maintains or improves performance in all, with notable gains in Seaquest and Alien; decorrelation is especially effective when using smaller batch sizes. The work demonstrates that network-wide online decorrelation can speed up gradient-based learning and enhance representation learning, offering a versatile approach compatible with CNNs and FC layers and with potential extensions to whitening, continuous actions, and broader RL domains.

Abstract

The effectiveness of credit assignment in reinforcement learning (RL) when dealing with high-dimensional data is influenced by the success of representation learning via deep neural networks, and has implications for the sample efficiency of deep RL algorithms. Input decorrelation has been previously introduced as a method to speed up optimization in neural networks, and has proven impactful in both efficient deep learning and as a method for effective representation learning for deep RL algorithms. We propose a novel approach to online decorrelation in deep RL based on the decorrelated backpropagation algorithm that seamlessly integrates the decorrelation process into the RL training pipeline. Decorrelation matrices are added to each layer, which are updated using a separate decorrelation learning rule that minimizes the total decorrelation loss across all layers, in parallel to minimizing the usual RL loss. We used our approach in combination with the soft actor-critic (SAC) method, which we refer to as decorrelated soft actor-critic (DSAC). Experiments on the Atari 100k benchmark with DSAC shows, compared to the regular SAC baseline, faster training in five out of the seven games tested and improved reward performance in two games with around 50% reduction in wall-clock time, while maintaining performance levels on the other games. These results demonstrate the positive impact of network-wide decorrelation in deep RL for speeding up its sample efficiency through more effective credit assignment.

Figures (10)

• Figure 1: DSAC speeds up training w.r.t. SAC in 5 of the 7 games tested for up to 76%, maintaining performance in all 7 and even improving performance in 2 games (Seaquest and Alien). Both algorithms use the best hyperparameter configuration for each game, with results averaged over 5 seeds. The shaded area shows the standard error of the mean. The games on which DSAC is faster than SAC are the games where DSAC uses a lower batch size (64) than SAC (256). For ChopperCommand, DSAC is slower due to higher batch size (256) against SAC (64). For Frostbite the time difference is small as SAC and DSAC use the same batch size (256).
• Figure 2: Decorrelation loss for the policy network (shown in log scale) constitutes the main difference across the compared algorithms, which is successfully minimized throughout the learning in DSAC, despite steady increase for the SAC baseline, where it reaches values beyond $10^6$. Plots are based on a SAC baseline trained with similar architectural conditions with DSAC, which differs from the baseline in the main results, to enable a fair comparison of the decorrelation losses.
• Figure 3: Search for the best hyperparameter configuration for the BP-based discrete SAC baseline. Values indicate the final reward average across five runs.
• Figure 4: Search for the best hyperparameter configuration for DSAC. Values indicate the final reward average across five runs. A decorrelation learning rate of 0.0 indicate no decorrelation for any of the SAC networks, hence the BP-based SAC baseline. A decorrelation learning rate more than 0.0 indicate decorrelation also in the soft-Q-networks, yet with a fixed learning rate of $1 \times 10^{-13}$, hence the DBP-based DSAC algorithm. Empty fields indicate NaNs.
• Figure 5: Reward curves per game for top configurations of SAC baseline vs. DSAC with decorrelation in the policy network only. Discrete SAC benchmark scores are based on christodoulou2019sacdiscrete, and are reported where available for a comparison. For the remaining games, scores of PPO for the Atari 100k benchmark, as reported by kaiser2020simple are provided, as PPO is another state-of-the-art actor-critic algorithm, albeit learning on-policy schulman2017proximal. Comparison to algorithms in the literature demonstrates that the results of our BP-based SAC baseline are aligned with the previously reported BP-based benchmarks for most of the games. Differences may be due to use of different seeds.