Neighboring State-based Exploration for Reinforcement Learning

TL;DR

This work tackles the exploration problem in reinforcement learning by introducing neighboring state-based exploration. It presents two model-free methods that search within a bounded region around the current state to guide exploratory actions, notably the Direct State Perturbation method (ρ-explore) and Change based exploration. Empirical results on Lunar Lander show that ρ-explore with intermittent scheduling can achieve approximately a 49.8% improvement in evaluation average return over a Double DQN baseline and improve stability. The approach offers a practical regularizer for exploration that leverages nearby states and may generalize to discrete-action, continuous-state tasks, with code provided for reproducibility.

Abstract

Reinforcement Learning is a powerful tool to model decision-making processes. However, it relies on an exploration-exploitation trade-off that remains an open challenge for many tasks. In this work, we study neighboring state-based, model-free exploration led by the intuition that, for an early-stage agent, considering actions derived from a bounded region of nearby states may lead to better actions when exploring. We propose two algorithms that choose exploratory actions based on a survey of nearby states, and find that one of our methods, $ρ$-explore, consistently outperforms the Double DQN baseline in an discrete environment by 49% in terms of Eval Reward Return.

Figures (6)

• Figure 1: Training Average Return with $\lambda=1$ step mini-rollouts; Upper: $max$, Lower: $mode$
• Figure 2: Training Average Return with $\lambda=10$ step mini-rollouts
• Figure 3: Eval Average Return with $\lambda=1$ step; $\rho$-explore interval=10
• Figure 4: Eval Average Return with $\lambda=1$ step; $\rho$-explore threshold=20; $\rho$-explore period=50, 100
• Figure 5: DQN Evaluation Average Return with $\epsilon = 0.75$