Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL

TL;DR

The paper analyzes why emergent exploration occurs in Single-Goal Contrastive RL (SGCRL) by showing that the actor optimizes an implicit, representation-based reward driven by $\psi$-similarity to the goal, while the critic shapes this implicit landscape through contrastive updates. It demonstrates that exploration and subsequent exploitation emerge from learning low-rank representations, not merely neural network capacity, and that the dynamics produce a curriculum of subgoals by pruning low-similarity regions before goal discovery and reinforcing high-similarity traces after discovery. The authors support their theory with tabular and continuous experiments across maze and Tower of Hanoi tasks, plus safety-oriented interventions that manipulate representations to guide behavior. The work provides a cognitive-science-inspired framework for understanding AI exploration and suggests practical pathways to safer deployment by controlling representation geometry rather than hand-crafted rewards. Overall, the findings offer a principled explanation for SGCRL’s robustness to distractors and their potential applicability to broader long-horizon, goal-directed RL settings.

Abstract

In this work, we take a first step toward elucidating the mechanisms behind emergent exploration in unsupervised reinforcement learning. We study Single-Goal Contrastive Reinforcement Learning (SGCRL), a self-supervised algorithm capable of solving challenging long-horizon goal-reaching tasks without external rewards or curricula. We combine theoretical analysis of the algorithm's objective function with controlled experiments to understand what drives its exploration. We show that SGCRL maximizes implicit rewards shaped by its learned representations. These representations automatically modify the reward landscape to promote exploration before reaching the goal and exploitation thereafter. Our experiments also demonstrate that these exploration dynamics arise from learning low-rank representations of the state space rather than from neural network function approximation. Our improved understanding enables us to adapt SGCRL to perform safety-aware exploration.

Figures (18)

• Figure 1: SGCRL exhibits emergent exploration on a range of tasks. But why?
• Figure 2: (a) The agent moves toward goal-like states (left), rules out non-goal states by assigning them low (green) $\psi$-similarity (center), and leaves a high (red) $\psi$-similarity trace along the successful trajectory once the goal is found (right). Solid lines indicate past paths; dashed lines indicate new paths. (b) Negative representations $\phi(s_i,a_i)$ and $\psi(s_f^{(j)})$ are pushed away from the common component $\psi(g)$, enhancing their contrast.
• Figure 3: Running SGCRL in a tabular Tower of Hanoi environment reveals distinct phases of learning. First majority success indicates the trial at which the majority of 5 evaluation runs succeeded. The 3 disk task is shown, but results generalize to 4/5 disks.
• Figure 4: (a) Initially all representations start close to $z$, later collapsing into a subspace orthogonal to $z$, while preserving local room-level structure. (b)$\psi$-similarity decreases along traversed, unsuccessful paths.
• Figure 5: SGCRL targets states with high $\psi$-similarity when initial sttate is perturbed.

Theorems & Definitions (10)

• Theorem 1
• Theorem 2: Informal
• Definition 1: Convergence
• Lemma 1: Positive examples have fully aligned representations
• proof
• Lemma 2
• proof
• proof
• Theorem 3: InfoNCE representations at equilibrium
• proof