DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization

TL;DR

Visual reinforcement learning from pixel inputs suffers from poor sample efficiency and seed-sensitive performance due to early inactive exploration. DrM addresses this by minimizing a formal inactivity metric, the dormant ratio $\beta_\tau$, through three mechanisms: a periodic weight perturbation, a dormancy-guided exploration schedule, and a dormancy-guided exploitation strategy that adapts based on $\beta$. The approach yields strong gains across three domains (DM Control Suite, MetaWorld, Adroit), including solving Dog and Manipulator tasks from pixels and demonstrating seed-robust performance with substantial reductions in required samples and improved asymptotic performance, while maintaining competitive wall-clock efficiency. These results introduce a robust, intrinsic activity signal to guide exploration, with potential extensions to unsupervised learning and discrete-action settings.

Abstract

Visual reinforcement learning (RL) has shown promise in continuous control tasks. Despite its progress, current algorithms are still unsatisfactory in virtually every aspect of the performance such as sample efficiency, asymptotic performance, and their robustness to the choice of random seeds. In this paper, we identify a major shortcoming in existing visual RL methods that is the agents often exhibit sustained inactivity during early training, thereby limiting their ability to explore effectively. Expanding upon this crucial observation, we additionally unveil a significant correlation between the agents' inclination towards motorically inactive exploration and the absence of neuronal activity within their policy networks. To quantify this inactivity, we adopt dormant ratio as a metric to measure inactivity in the RL agent's network. Empirically, we also recognize that the dormant ratio can act as a standalone indicator of an agent's activity level, regardless of the received reward signals. Leveraging the aforementioned insights, we introduce DrM, a method that uses three core mechanisms to guide agents' exploration-exploitation trade-offs by actively minimizing the dormant ratio. Experiments demonstrate that DrM achieves significant improvements in sample efficiency and asymptotic performance with no broken seeds (76 seeds in total) across three continuous control benchmark environments, including DeepMind Control Suite, MetaWorld, and Adroit. Most importantly, DrM is the first model-free algorithm that consistently solves tasks in both the Dog and Manipulator domains from the DeepMind Control Suite as well as three dexterous hand manipulation tasks without demonstrations in Adroit, all based on pixel observations.

Figures (24)

• Figure 2: (Dormant ratio of a DrQ-v2 agent trained on Hopper Hop task during the first 1M frames): Interestingly, we find that with a declining dormant ratio, the agent incrementally acquires action capabilities. Even though the reward stays minimal during this phase, the dormant ratio provides a more insightful gauge of the agent's initial learning progress than the reward does.
• Figure 3: Visualization of the awaken exploration scheduler as a function of the dormant ratio $\beta$
• Figure 4: Visualization of exploitation hyperparameter as a function of the dormant ratio $\beta$
• Figure 5: DeepMind Control Suite
• Figure 6: MetaWorld

Theorems & Definitions (2)

• Definition 2.1
• Definition 2.2