Explicit-Implicit Subgoal Planning for Long-Horizon Tasks with Sparse Reward

TL;DR

The paper tackles long-horizon robotic tasks hindered by sparse rewards by introducing Explicit-Implicit Subgoal Planning (EISP), a three-component framework that generates feasible subgoals via a Hybrid Subgoal Generator (CVAE-based explicit encoder and implicit decoder), ensures feasibility with a Hindsight Sampler, and selects optimal subgoals using a Value Selector grounded in a Universal Value Function Approximator. It integrates with Soft Actor-Critic and Hindsight Experience Replay to train a robust goal-conditioned policy across four manipulation tasks, both in simulation and on real hardware. Key contributions include the hybrid subgoal generator, an offline-data-driven mechanism for feasible subgoals, and a subgoal ranking strategy that improves long-horizon performance, validated by extensive experiments and ablation studies. The approach advances practical divide-and-conquer planning in robotics, enabling more reliable execution of complex multi-step tasks with sparse rewards and demonstrating strong transfer from simulation to real-world settings.

Abstract

The challenges inherent in long-horizon tasks in robotics persist due to the typical inefficient exploration and sparse rewards in traditional reinforcement learning approaches. To address these challenges, we have developed a novel algorithm, termed Explicit-Implicit Subgoal Planning (EISP), designed to tackle long-horizon tasks through a divide-and-conquer approach. We utilize two primary criteria, feasibility and optimality, to ensure the quality of the generated subgoals. EISP consists of three components: a hybrid subgoal generator, a hindsight sampler, and a value selector. The hybrid subgoal generator uses an explicit model to infer subgoals and an implicit model to predict the final goal, inspired by way of human thinking that infers subgoals by using the current state and final goal as well as reason about the final goal conditioned on the current state and given subgoals. Additionally, the hindsight sampler selects valid subgoals from an offline dataset to enhance the feasibility of the generated subgoals. While the value selector utilizes the value function in reinforcement learning to filter the optimal subgoals from subgoal candidates. To validate our method, we conduct four long-horizon tasks in both simulation and the real world. The obtained quantitative and qualitative data indicate that our approach achieves promising performance compared to other baseline methods. These experimental results can be seen on the website \url{https://sites.google.com/view/vaesi}.

Figures (17)

• Figure 1: (a) EISP infers multiple subgoals at different task stages. Green and red lines represent possible subgoal sequences. (b) We leverage the explicit model to infer subgoals and the implicit model to predict the final goal.
• Figure 2: Examples of feasible and optimal subgoals in the OpenDrawer environment are depicted.
• Figure 3: The Explicit-Implicit Subgoal Planning (EISP) algorithm consists of three main components: a Hybrid Subgoal Generator, a Hindsight Sampler, and a Value Selector. The subgoal generator takes as input the current state and the desired goal and outputs subgoals to accomplish the long-horizon task. The Value Selector and the Hindsight Sampler are utilized to ensure that the subgoals are optimal and feasible, respectively.
• Figure 4: Details of our proposed subgoal inference method. It inherits the variational autoencoder structure, where the encoder generates the subgoals by taking the current state and the final goal as inputs, and the decoder generates a reconstructed final goal conditioned on the current state and the subgoal.
• Figure 5: Hindsight Sampler to guide the generator to produce feasible subgoals.

Theorems & Definitions (3)

• Definition 4.1: Feasibility
• Definition 4.2: Optimality
• proof