Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slow

TL;DR

This paper tackles the computational bottleneck of applying recurrent models in off-policy image-based RL by introducing Recurrent Integration via Simplified Encodings (RISE), a dual-stream framework that uses fixed non-learnable embeddings alongside a learnable encoder to supply long-term context to a recurrent module. By precomputing and storing non-learnable embeddings in the replay buffer, RISE dramatically reduces encoder passes per update, enabling long context with modest overhead ($k=160$) and strong performance across diverse domains, notably improving Beyond The Rainbow (BTR) on Atari by a substantial margin (e.g., $35.6\%$ human-normalized IQM). The authors provide extensive ablations on embedding choices, combination methods, context length, and sequence architectures, and show that two-stream LSTMs outperform alternative designs and single-stream baselines while maintaining favorable walltime. The work broadens access to high-performing recurrent off-policy RL and opens avenues for integrating pretrained representations in compute-constrained settings, with implications for other modalities beyond vision.

Abstract

Recurrent off-policy deep reinforcement learning models achieve state-of-the-art performance but are often sidelined due to their high computational demands. In response, we introduce RISE (Recurrent Integration via Simplified Encodings), a novel approach that can leverage recurrent networks in any image-based off-policy RL setting without significant computational overheads via using both learnable and non-learnable encoder layers. When integrating RISE into leading non-recurrent off-policy RL algorithms, we observe a 35.6% human-normalized interquartile mean (IQM) performance improvement across the Atari benchmark. We analyze various implementation strategies to highlight the versatility and potential of our proposed framework.

Figures (36)

• Figure 1: Interquartile mean human-normalized performance showing the BTR algorithm with and without our proposed method, Recurrent Integration via Simplified Encodings (RISE), on the Atari-55 benchmark. All algorithms use 200 million frames, and shaded areas show 95% bootstrapped confidence intervals. Dreamer-v3, Rainbow and DQN refer to hafner2301masteringhessel2018rainbowmnih2015human respectively.
• Figure 2: Diagram comparing different RL architectures, with and without a recurrent model. While we use a sigmoid function and multiplication, following the linear upscaling layer, the combination mechanism is an open design choice. Total Convolutional Neural Network (CNN) passes are the requirement to compute a single Q-value.
• Figure 3: Performance of Vectorized Rainbow and BTR with typical LSTMS, RISE, and no recurrent model. All models were run for 200M frames. Experiments were performed on the Atari-5 subset, using a desktop PC with an RTX4090 GPU. Areas show 95% bootstrapped confidence intervals over 3 seeds. For typical LSTMs, runs were performed using a context length of 20 instead of RISE's 160 due requiring $>$ 300 walltime hours. For compute resource details, see Appendix \ref{['app:compute_resource']}.
• Figure 4: Box plot performance on Atari-57 of BTR+RISE (3 seeds) against other algorithms . Note scores for all algorithms use final scores at 200M, not best scores. Areas show 95% bootstrapped confidence intervals over seeds. Labels show the walltime for 200M frames, alongside the GPU used.
• Figure 5: Percentage human normalized improvement of BTR + RISE over BTR in each of the environments in Atari-57. Error bars show 95% bootstrapped confidence intervals over 3 seeds.