Dissecting Deep RL with High Update Ratios: Combatting Value Divergence

TL;DR

This paper tackles sample-efficient deep RL under high update-to-data ratios ($UTD$), where primacy bias has been observed. It identifies value divergence of the $Q$-function, amplified by out-of-distribution action predictions and optimizer momentum, as a core learning obstacle rather than simple early overfitting. The authors propose unit-ball feature normalization (OFN) applied to the critic encoder to decouple $Q$-scale from early layers, enabling stable learning without resets and achieving strong results on the dm_control suite and challenging dog tasks, competitive with model-based approaches like TD-MPC2. They provide extensive analysis of regularization and other failure modes, showing OFN is simple, effective, but that actor and exploration issues remain open. Overall, the work suggests optimization-driven divergence as a central factor in high-$UTD$ RL and offers a practical technique to enable robust learning in such regimes.

Abstract

We show that deep reinforcement learning algorithms can retain their ability to learn without resetting network parameters in settings where the number of gradient updates greatly exceeds the number of environment samples by combatting value function divergence. Under large update-to-data ratios, a recent study by Nikishin et al. (2022) suggested the emergence of a primacy bias, in which agents overfit early interactions and downplay later experience, impairing their ability to learn. In this work, we investigate the phenomena leading to the primacy bias. We inspect the early stages of training that were conjectured to cause the failure to learn and find that one fundamental challenge is a long-standing acquaintance: value function divergence. Overinflated Q-values are found not only on out-of-distribution but also in-distribution data and can be linked to overestimation on unseen action prediction propelled by optimizer momentum. We employ a simple unit-ball normalization that enables learning under large update ratios, show its efficacy on the widely used dm_control suite, and obtain strong performance on the challenging dog tasks, competitive with model-based approaches. Our results question, in parts, the prior explanation for sub-optimal learning due to overfitting early data.

Figures (18)

• Figure 1: Return, in-distribution Q-values and Adam optimizer moments during priming for different lengths. Dotted lines correspond to end of priming. More priming leads to lower return and larger Q-value and optimizer divergence.
• Figure 2: Priming with SAC and action regularization during priming. The latter lowers divergence.
• Figure 3: Return and Q-values of priming runs with weight decay and dropout. Results indicate that both regularizations mitigate priming to some extent but not sufficiently.
• Figure 4: In-distribution Q-values and critic loss of five SAC seeds on the humanoid-run task using $\mathrm{UTD}=32$. Values diverge at arbitrary time-points, not only during the beginning. Loss mirrors Q-value divergence.
• Figure 5: (Left) Return and (Right) Q-values comparing SGD result and OFN when priming for 100K steps. OFN obtains returns close to that of the well-trained SGD agent and learns an appropriate Q-value scale correctly.