Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy

Hongming Li, Shujian Yu, Bin Liu, Jose C. Principe

TL;DR

ELEMENT tackles the lack of extrinsic rewards by maximizing both episodic and lifelong state entropy in RL. It introduces a fixed encoder, a Markovian episodic proxy reward $r_{ep}$ for trajectory entropy, and a fast $k$NN-graph based lifelong reward $r_{l}$, combining them as $r_i(\mathbf{s}) = r_{ep}(\mathbf{s}) + \beta r_{l}(\mathbf{s})$. Empirically, it demonstrates stronger episodic entropy maximization, broader exploration, effective offline RL data collection, and improved task-agnostic transfer, while revealing practical limitations related to fixed representations and episode-length trade-offs. The framework offers practical benefits for unsupervised pre-training and data-driven RL, with potential for improved sample efficiency and downstream performance across robotics and simulated domains. Overall, ELEMENT advances intrinsic exploration by balancing rapid episodic diversity with long-horizon state coverage, enabling robust, scalable, and transferable exploration policies.

Abstract

This paper proposes \emph{Episodic and Lifelong Exploration via Maximum ENTropy} (ELEMENT), a novel, multiscale, intrinsically motivated reinforcement learning (RL) framework that is able to explore environments without using any extrinsic reward and transfer effectively the learned skills to downstream tasks. We advance the state of the art in three ways. First, we propose a multiscale entropy optimization to take care of the fact that previous maximum state entropy, for lifelong exploration with millions of state observations, suffers from vanishing rewards and becomes very expensive computationally across iterations. Therefore, we add an episodic maximum entropy over each episode to speedup the search further. Second, we propose a novel intrinsic reward for episodic entropy maximization named \emph{average episodic state entropy} which provides the optimal solution for a theoretical upper bound of the episodic state entropy objective. Third, to speed the lifelong entropy maximization, we propose a $k$ nearest neighbors ($k$NN) graph to organize the estimation of the entropy and updating processes that reduces the computation substantially. Our ELEMENT significantly outperforms state-of-the-art intrinsic rewards in both episodic and lifelong setups. Moreover, it can be exploited in task-agnostic pre-training, collecting data for offline reinforcement learning, etc.

ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy

TL;DR

ELEMENT tackles the lack of extrinsic rewards by maximizing both episodic and lifelong state entropy in RL. It introduces a fixed encoder, a Markovian episodic proxy reward for trajectory entropy, and a fast NN-graph based lifelong reward , combining them as . Empirically, it demonstrates stronger episodic entropy maximization, broader exploration, effective offline RL data collection, and improved task-agnostic transfer, while revealing practical limitations related to fixed representations and episode-length trade-offs. The framework offers practical benefits for unsupervised pre-training and data-driven RL, with potential for improved sample efficiency and downstream performance across robotics and simulated domains. Overall, ELEMENT advances intrinsic exploration by balancing rapid episodic diversity with long-horizon state coverage, enabling robust, scalable, and transferable exploration policies.

Abstract

This paper proposes \emph{Episodic and Lifelong Exploration via Maximum ENTropy} (ELEMENT), a novel, multiscale, intrinsically motivated reinforcement learning (RL) framework that is able to explore environments without using any extrinsic reward and transfer effectively the learned skills to downstream tasks. We advance the state of the art in three ways. First, we propose a multiscale entropy optimization to take care of the fact that previous maximum state entropy, for lifelong exploration with millions of state observations, suffers from vanishing rewards and becomes very expensive computationally across iterations. Therefore, we add an episodic maximum entropy over each episode to speedup the search further. Second, we propose a novel intrinsic reward for episodic entropy maximization named \emph{average episodic state entropy} which provides the optimal solution for a theoretical upper bound of the episodic state entropy objective. Third, to speed the lifelong entropy maximization, we propose a nearest neighbors (NN) graph to organize the estimation of the entropy and updating processes that reduces the computation substantially. Our ELEMENT significantly outperforms state-of-the-art intrinsic rewards in both episodic and lifelong setups. Moreover, it can be exploited in task-agnostic pre-training, collecting data for offline reinforcement learning, etc.

Paper Structure

This paper contains 24 sections, 2 theorems, 24 equations, 12 figures, 8 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Backgrounds
  3. Entropy Estimators
  4. Reinforcement learning for exploration with maximum state entropy
  5. Parametric Methods for Exploration
  6. Intrinsic Reward with ELEMENT
  7. Fixed neural encoder $f: \mathcal{O} \rightarrow \mathcal{S}$.
  8. Episodic intrinsic reward.
  9. Lifelong intrinsic reward.
  10. Empirical Analysis
  11. How Does ELEMENT Drive the Agent?
  12. Can ELEMENT Maximize Episodic State Entropy?
  13. Can ELEMENT Explore New States Efficiently?
  14. Can the Samples Collected by ELEMENT Agents be Used for Offline learning?
  15. Can ELEMENT agents Conduct Task-Agnostic Exploration for Downstream Tasks?
  16. ...and 9 more sections

Key Result

Proposition 1

Assume all episodes have the same length $T_{\tau}$, we define an upper bound of Eq. (loss_reward) given by the Monte-Carlo estimation ren2021learning: where $V_{\mathbf{s} \in \tau}(r_{ep}(\mathbf{s})) \geq 0$ denotes the expected variance of rewards within the episodes. The optimal solution $r^{*}_{ep}$ to minimize $\tilde{\mathcal{L}}(r_{ep})$ is given by:

Figures (12)

  • Figure 1: Overview of our approach. The life long intrinsic reward of a state $\mathbf{s}$ (the green node) is computed as the distance to its $k$ nearest neighbors, measured in representations space obtained from a fixed neural encoder. The episodic intrinsic reward is calculated as the average state entropy of these episodes including the state. The lifelong reward is then combined with episodic reward. A separate DRL is introduced for a policy that maximizes expected reward.
  • Figure 2: Episodic entropy maximization encourages movement in a single direction to capture more distinct states per episode. Deviating from this direction will result in smaller distances to the previously visited states within the same episode. Lifelong motivation discourages revisiting episodes, hindering a continual exploration towards one direction. ELEMENTs shares merits of both, promoting comprehensive exploration without confinement to one direction.
  • Figure 3: The $k$NN graph search and online update algorithms on simple nearest neighbor graphs. In searching phase, the algorithm search by moving to the neighbor that is closest to the query. After that, neighbors obtained by the search phase are used as starting points to update existing nodes for which the new point is now a nearest neighbor.
  • Figure 4: Trajectories obtained by executing policies at different checkpoints (depicted by different colors) on Walker2D, Ant and Mario. At the first checkpoint, the agent quickly converges to a policy which produces similar episodes with many different states. When the learning process continues and the $k$NN graph gradually updates, agents slowly try to stay away from visited episodes but still make states within an episode different.
  • Figure 5: (a) The learning curve given by parametric modeling of the episodic reward $r_{ep}(\mathbf{s})$ using a DNN. (b) Predicted episodic state entropy using $r_{ep}(\mathbf{s})$ in ELEMENT.
  • ...and 7 more figures

Theorems & Definitions (2)

  • Proposition 1
  • Proposition 2