A Temporally Correlated Latent Exploration for Reinforcement Learning

TL;DR

The paper tackles exploration in deep reinforcement learning under sparse rewards and noisy environments by introducing TeCLE, a method that defines intrinsic rewards from an action-conditioned latent space learned via a CVAE and injects temporal correlation into the intrinsic signal. This design yields robustness to Noisy TV and stochasticity, demonstrated on Minigrid and Stochastic Atari, with system behavior tunable through a colored-noise parameter $\beta$. Key findings show that action conditioning and intrinsic rewards without extrinsic rewards can enable learning in challenging settings, and that the choice of temporal correlation (e.g., $\beta<0$ for robustness to noise, $\beta>0$ for broader exploration) significantly shapes exploration dynamics. Overall, TeCLE provides a principled, robust intrinsic-motivation framework for hard exploration tasks with potential applicability to other intrinsic-motivation schemes.

Abstract

Efficient exploration remains one of the longstanding problems of deep reinforcement learning. Instead of depending solely on extrinsic rewards from the environments, existing methods use intrinsic rewards to enhance exploration. However, we demonstrate that these methods are vulnerable to Noisy TV and stochasticity. To tackle this problem, we propose Temporally Correlated Latent Exploration (TeCLE), which is a novel intrinsic reward formulation that employs an action-conditioned latent space and temporal correlation. The action-conditioned latent space estimates the probability distribution of states, thereby avoiding the assignment of excessive intrinsic rewards to unpredictable states and effectively addressing both problems. Whereas previous works inject temporal correlation for action selection, the proposed method injects it for intrinsic reward computation. We find that the injected temporal correlation determines the exploratory behaviors of agents. Various experiments show that the environment where the agent performs well depends on the amount of temporal correlation. To the best of our knowledge, the proposed TeCLE is the first approach to consider the action conditioned latent space and temporal correlation for curiosity-driven exploration. We prove that the proposed TeCLE can be robust to the Noisy TV and stochasticity in benchmark environments, including Minigrid and Stochastic Atari.

Figures (16)

• Figure 1: Architecture of proposed TeCLE. (Part A) Feature Embedding learns the state representations $\phi(s_t)$ and $\phi(s_{t+1})$ using embedding network $f_{\theta}$ and inverse network $g_{\theta}$; (Part B) Action-Conditioned Latent Exploration computes intrinsic reward $r^{\text{i}}_{t}$ using the reconstructed state representation $\hat{\phi}(s_{t+1})$ and the $\phi(s_{t+1})$; (Part C) Colored Noise injects $\varepsilon_{t+1}$ when sampling the latent representation $z_{t+1}$ of Part B.
• Figure 2: Normalized average returns across environments in Table \ref{['tab:beta']}. The error bars show the mean (dots) and standard error (upper and lower bounds) of the normalized average returns according to $\beta$.
• Figure 3: Visualized state coverages in DoorKey$16\times16$ and Empty$16\times16$ without Noisy TV. In DoorKey$16\times16$, only TeCLE with red ($\beta=2.0$) and blue ($\beta=-1.0$) noises can solve the tasks and learned the optimal policy for exploration. It seems that blue noise ($\beta=-1.0$) encourages agents to exploit more than explore compared to red noise. We think that TeCLE encourages the agent to explore more than exploit compared to low $\beta$, which is similar to the studies in eberhard2023pinkhollenstein2024colored.
• Figure 4: Comparison on Minigrid environments with Noisy TV. In DoorKey, other methods except for TeCLE failed to avoid the Noisy TV. Generally, red noise ($\beta=2.0$) was more effective than other colored noises.
• Figure 5: Comparison on Minigrid environments without Noisy TV. Only TeCLE can show convergence in both LavaCrossingS11N5 (large state-space) and KeyCorridorS3R3 (hard task).