Episodic Novelty Through Temporal Distance

TL;DR

Episodic Novelty Through Temporal Distance (ETD) addresses exploration in sparse-reward Contextual MDPs by learning a robust temporal distance as a state-similarity metric via contrastive learning. The intrinsic reward is $b_{\mathrm{ETD}}(s_t)=\min_{k< t} d_{\phi}(s_k,s_t)$, where $d_{\phi}$ is a learned successor-distance-based quasimetric, enabling efficient, episodic exploration without expensive world-model rollouts. ETD employs an MRN-based architecture and InfoNCE-based training to estimate temporal distances, and demonstrates strong improvements in sample efficiency across MiniGrid, Crafter, and MiniWorld benchmarks, especially under noise and high-dimensional observations. The work shows ETD outperforms state-of-the-art episodic and global intrinsic rewards (e.g., NovelD, DEIR, EC, NGU) and discusses potential extensions to combine temporal-distance bonuses with global signals, along with limitations related to POMDP-like effects and non-ergodic settings.

Abstract

Exploration in sparse reward environments remains a significant challenge in reinforcement learning, particularly in Contextual Markov Decision Processes (CMDPs), where environments differ across episodes. Existing episodic intrinsic motivation methods for CMDPs primarily rely on count-based approaches, which are ineffective in large state spaces, or on similarity-based methods that lack appropriate metrics for state comparison. To address these shortcomings, we propose Episodic Novelty Through Temporal Distance (ETD), a novel approach that introduces temporal distance as a robust metric for state similarity and intrinsic reward computation. By employing contrastive learning, ETD accurately estimates temporal distances and derives intrinsic rewards based on the novelty of states within the current episode. Extensive experiments on various benchmark tasks demonstrate that ETD significantly outperforms state-of-the-art methods, highlighting its effectiveness in enhancing exploration in sparse reward CMDPs.

Figures (20)

• Figure 1: Training curves in Minigrid-DooKey-16x16 (w/w.o. noise).
• Figure 2: Distance from $$ to all other states in a 17x17 SpiralMaze. Darker colors indicate greater distance. (Left) Euclidean distance of embeddings trained by inverse dynamics. (Center) Likelihood estimation of easy transitions (EC). (Right) The learned temporal distance (Ours).
• Figure 3: Overview of ETD. ETD encourages visits to temporally distant states from episodic memory. Temporal distance can be learned through contrastive learning when positive samples consist of a state and its geometrically distributed future state, with the erengy function parameterized as a potential network minus a quasimetric network. The intrinsic reward is then derived from the minimum temporal distance between the current state and the states stored in episodic memory.
• Figure 4: Rendering of the environments used in this work.
• Figure 5: Training performance of ETD and the baselines on 8 most challenging Minigrid environments. The x-axis represents the environment steps. All the results are averaged across 5 seeds.

Theorems & Definitions (8)

• Proposition 1
• proof
• Corollary 1
• proof
• Proposition 2
• proof
• Proposition 3
• proof