Incorporating Spatial Information into Goal-Conditioned Hierarchical Reinforcement Learning via Graph Representations
Shuyuan Zhang, Zihan Wang, Xiao-Wen Chang, Doina Precup
TL;DR
The paper tackles the inefficiency of goal-conditioned hierarchical RL by introducing G4RL, which embeds spatial information through a graph encoder–decoder and an online state graph. By constructing and updating a state graph during exploration, and learning subgoal representations that respect connectivity, the approach provides intrinsic rewards at both the high and low levels to guide exploration and execution. The method is designed to be compatible with existing GCHRL algorithms and shows substantial improvements in convergence speed and success rates across dense and sparse reward Ant environments, including image-based state experiments. While effective in symmetric and reversible transition settings, the work also investigates adaptive training and speed-accuracy trade-offs and highlights potential future work on automatic hyperparameter tuning and transfer of graph knowledge to new tasks.
Abstract
The integration of graphs with Goal-conditioned Hierarchical Reinforcement Learning (GCHRL) has recently gained attention, as intermediate goals (subgoals) can be effectively sampled from graphs that naturally represent the overall task structure in most RL tasks. However, existing approaches typically rely on domain-specific knowledge to construct these graphs, limiting their applicability to new tasks. Other graph-based approaches create graphs dynamically during exploration but struggle to fully utilize them, because they have problems passing the information in the graphs to newly visited states. Additionally, current GCHRL methods face challenges such as sample inefficiency and poor subgoal representation. This paper proposes a solution to these issues by developing a graph encoder-decoder to evaluate unseen states. Our proposed method, Graph-Guided sub-Goal representation Generation RL (G4RL), can be incorporated into any existing GCHRL method when operating in environments with primarily symmetric and reversible transitions to enhance performance across this class of problems. We show that the graph encoder-decoder can be effectively implemented using a network trained on the state graph generated during exploration. Empirical results indicate that leveraging high and low-level intrinsic rewards from the graph encoder-decoder significantly enhances the performance of state-of-the-art GCHRL approaches with an extra small computational cost in dense and sparse reward environments.