Search Inspired Exploration in Reinforcement Learning

TL;DR

SIERL tackles sparse-reward reinforcement learning by introducing frontier-guided exploration through sub-goal setting. It couples a two-phase exploration strategy with frontier extraction from the replay buffer and a sub-goal selection mechanism that optimizes a composite score of novelty, cost-to-come, and cost-to-go via softmin sampling. The method demonstrates competitive main-goal achievement and superior generalization to arbitrary states in discrete Hallway, FourRooms, and BugTrap environments, with ablations confirming the importance of frontier-based prioritization and controlled phase switching. These results suggest that deliberate, frontier-driven curricula can improve data efficiency and robustness in goal-conditioned RL without relying on manually crafted curricula or purely intrinsic rewards.

Abstract

Exploration in environments with sparse rewards remains a fundamental challenge in reinforcement learning (RL). Existing approaches such as curriculum learning and Go-Explore often rely on hand-crafted heuristics, while curiosity-driven methods risk converging to suboptimal policies. We propose Search-Inspired Exploration in Reinforcement Learning (SIERL), a novel method that actively guides exploration by setting sub-goals based on the agent's learning progress. At the beginning of each episode, SIERL chooses a sub-goal from the \textit{frontier} (the boundary of the agent's known state space), before the agent continues exploring toward the main task objective. The key contribution of our method is the sub-goal selection mechanism, which provides state-action pairs that are neither overly familiar nor completely novel. Thus, it assures that the frontier is expanded systematically and that the agent is capable of reaching any state within it. Inspired by search, sub-goals are prioritized from the frontier based on estimates of cost-to-come and cost-to-go, effectively steering exploration towards the most informative regions. In experiments on challenging sparse-reward environments, SIERL outperforms dominant baselines in both achieving the main task goal and generalizing to reach arbitrary states in the environment.

Figures (11)

• Figure 1: The MiniGrid room variants.
• Figure 2: Main-goal (top row) and random-goal (bottom row) performance for the Hallway variant in columns. SIERL achieves a remarkable performance for both criteria at the same time, matched by no other method.
• Figure 3: Success rate for reaching: the main goal (top), and random goals (bottom) for ablated variants, in 6-steps Hallway (left) and FourRooms (right). Most notably, removing frontier filtering or prioritization worsens SIERL's main-goal success, while removing early switching shows a smaller negative influence.
• Figure 4: Success rate for the main-goal and random-goals in 4-step long Hallway for SIERL with varying familiarity thresholds, $F_\pi^{\mathrm{thr}}$.
• Figure 5: Success rate for the main-goal and random-goals in 4-step long Hallway for SIERL with varying novelty weights, $w_{\mathrm{n}}$.

Theorems & Definitions (2)

• Definition 4.1: State Familiarity
• Definition 4.2: Trajectory Familiarity