Learning Without Time-Based Embodiment Resets in Soft-Actor Critic

TL;DR

This paper tackles the reliance on episode terminations and embodiment resets in reinforcement learning by proposing continuing SAC, a continuing-task variant of Soft Actor-Critic with minimal changes. It shows that with reward-function adjustments, continuing SAC can rival episodic SAC and exhibit reduced sensitivity to the discount factor $\gamma$, but learns without resets can suffer from poor exploration due to high variance in the critic's outputs. The authors identify the high variance of Q-values as the likely root cause of exploration failure and demonstrate effective remedies, including layer normalization and dynamic entropy tuning (tilde-$\alpha$-toggle), which recover performance in several simulated tasks and a real-robot vision task, sometimes outperforming baseline exploration methods like RND. Overall, the work advances reset-free reinforcement learning by diagnosing exploration bottlenecks and offering practical interventions that improve robustness across diverse continuous-control tasks.

Abstract

When creating new reinforcement learning tasks, practitioners often accelerate the learning process by incorporating into the task several accessory components, such as breaking the environment interaction into independent episodes and frequently resetting the environment. Although they can enable the learning of complex intelligent behaviors, such task accessories can result in unnatural task setups and hinder long-term performance in the real world. In this work, we explore the challenges of learning without episode terminations and robot embodiment resets using the Soft Actor-Critic (SAC) algorithm. To learn without terminations, we present a continuing version of the SAC algorithm and show that, with simple modifications to the reward functions of existing tasks, continuing SAC can perform as well as or better than episodic SAC while reducing the sensitivity of performance to the value of the discount rate $γ$. On a modified Gym Reacher task, we investigate possible explanations for the failure of continuing SAC when learning without embodiment resets. Our results suggest that embodiment resets help with exploration of the state space in the SAC algorithm, and removing embodiment resets can lead to poor exploration of the state space and failure of or significantly slower learning. Finally, on additional simulated tasks and a real-robot vision task, we show that increasing the entropy of the policy when performance trends worse or remains static is an effective intervention for recovering the performance lost due to not using embodiment resets.

Figures (24)

• Figure 1: Although continuing SAC can achieve higher average reward than episodic SAC in most runs (a), it obtains much lower returns of less than 1000 (b). Modifying the reward function of Hopper by removing the constant term and adding a penalty for falling enables continuing SAC to perform as well as episodic SAC (c).
• Figure 2: Continuing SAC performs similarly to episodic SAC in all tested environments. All environments use resets, which mark the end of an episode in episodic SAC and a state transition to the state after reset in continuing SAC.
• Figure 3: The performance of continuing SAC is not sensitive to the value of its hyper-parameter $\alpha_{\bar{R}}$ (top) and is less sensitive to the value of $\gamma$ than episodic SAC, particularly for $\gamma$ values close to one (bottom).
• Figure 4: Not using time-based resets results in failing to learn or significantly slower learning in most runs (a) and significantly lower variance in the visited states and poor exploration of the state space (b).
• Figure 5: a) All three curves use continuing SAC without the entropy term in the policy objective. In contrast to a nonstationary Q function, fixed and learning Q functions lead to zero average reward for most of the run, indicating poor exploration, which suggests that adequate exploration of the state space in the SAC algorithm relies on a changing Q function. b) With all values except the default $\bar{\mathcal{H}}=-2$, performance is higher using the entropy-only objective $J^{\mathcal{H}} (\theta)$ compared to the baseline original objective $J (\theta)$, which indicates that the entropy term alone of the policy objective of SAC is not responsible for the poor exploration when learning without resets.