Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Goal-Space Planning with Subgoal Models

Chunlok Lo, Kevin Roice, Parham Mohammad Panahi, Scott Jordan, Adam White, Gabor Mihucz, Farzane Aminmansour, Martha White

TL;DR

Goal-Space Planning (GSP) introduces subgoal-conditioned, local models to enable efficient background planning in online RL. It constructs an abstract MDP over a predefined set of subgoals, performs value iteration to compute subgoal values, and projects these values back to the original state space via potential-based shaping to guide learning without altering the optimal policy. Across multiple domains, GSP accelerates value propagation and learning, showing robustness to moderate model inaccuracies and a clear dependence on subgoal placement. Unlike Dyna-style approaches that learn full transition dynamics, GSP focuses planning on a compact, temporally abstract space for computational efficiency and faster adaptation.

Abstract

This paper investigates a new approach to model-based reinforcement learning using background planning: mixing (approximate) dynamic programming updates and model-free updates, similar to the Dyna architecture. Background planning with learned models is often worse than model-free alternatives, such as Double DQN, even though the former uses significantly more memory and computation. The fundamental problem is that learned models can be inaccurate and often generate invalid states, especially when iterated many steps. In this paper, we avoid this limitation by constraining background planning to a set of (abstract) subgoals and learning only local, subgoal-conditioned models. This goal-space planning (GSP) approach is more computationally efficient, naturally incorporates temporal abstraction for faster long-horizon planning and avoids learning the transition dynamics entirely. We show that our GSP algorithm can propagate value from an abstract space in a manner that helps a variety of base learners learn significantly faster in different domains.

Goal-Space Planning with Subgoal Models

TL;DR

Goal-Space Planning (GSP) introduces subgoal-conditioned, local models to enable efficient background planning in online RL. It constructs an abstract MDP over a predefined set of subgoals, performs value iteration to compute subgoal values, and projects these values back to the original state space via potential-based shaping to guide learning without altering the optimal policy. Across multiple domains, GSP accelerates value propagation and learning, showing robustness to moderate model inaccuracies and a clear dependence on subgoal placement. Unlike Dyna-style approaches that learn full transition dynamics, GSP focuses planning on a compact, temporally abstract space for computational efficiency and faster adaptation.

Abstract

This paper investigates a new approach to model-based reinforcement learning using background planning: mixing (approximate) dynamic programming updates and model-free updates, similar to the Dyna architecture. Background planning with learned models is often worse than model-free alternatives, such as Double DQN, even though the former uses significantly more memory and computation. The fundamental problem is that learned models can be inaccurate and often generate invalid states, especially when iterated many steps. In this paper, we avoid this limitation by constraining background planning to a set of (abstract) subgoals and learning only local, subgoal-conditioned models. This goal-space planning (GSP) approach is more computationally efficient, naturally incorporates temporal abstraction for faster long-horizon planning and avoids learning the transition dynamics entirely. We show that our GSP algorithm can propagate value from an abstract space in a manner that helps a variety of base learners learn significantly faster in different domains.
Paper Structure (47 sections, 5 theorems, 52 equations, 24 figures, 7 algorithms)

This paper contains 47 sections, 5 theorems, 52 equations, 24 figures, 7 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Goal-Space Planning at a High-Level
  4. Motivating Experiments for GSP
  5. GSP on Propagating Value
  6. GSP in Larger State Spaces
  7. Goal-Space Planning in Detail
  8. Defining Subgoals
  9. Defining Subgoal-Conditioned Models
  10. Goal-Space Planning with Subgoal-Conditioned Models
  11. Using Subgoal Values to Update the Policy
  12. Putting it All Together: The Full Goal-Space Planning Algorithm
  13. Experiments to understand GSP more deeply
  14. GSP with Deep Reinforcement Learning
  15. Robustness to Accuracy of the Learned Models
  16. ...and 32 more sections

Key Result

Lemma 1

Assume that we have a deterministic MDP, deterministic policy $\pi$, $\gamma_c < 1$, a discrete set of subgoals $\mathcal{G} \subset \mathcal{S}$, and that we iteratively update $v_t \in \mathbb{R}^{|\bar{\mathcal{G}}|}$ with the dynamic programming update for all $g \in \mathcal{G}$, starting from an arbitrary (finite) initialization $v_0 \in \mathbb{R}^{|\bar{\mathcal{G}}|}$, with $v_t(s_{\text

Figures (24)

  • Figure 1: GSP in the PinBall domain. The agent begins with a set of subgoals (denoted in teal) and learns a set of subgoal-conditioned models. (Abstraction) Using these models, the agent forms an abstract MDP where the states are subgoals with options to reach each subgoal as actions. (Planning) The agent plans in this abstract MDP to quickly learn the values of these subgoals. (Projection) Using learned subgoal values, the agent obtains approximate values of states based on nearby subgoals and their values. These quickly updated approximate values are then used to speed up learning.
  • Figure 2: The FourRooms domain. The blue square is the initial state, green square the goal state, and red boxes the subgoals. A subgoal's initiation set contains the states in any room connected to that subgoal.
  • Figure 3: These four plots show the action values after a single episode of updates for Sarsa with and without GSP and eligibility traces, i.e., $\lambda = 0.9$. Each algorithm's update is simulated from the same data collected from a uniform random policy. Each state (square) is made up of four triangles representing each of the four available actions. White squares represent states not visited in the episode.
  • Figure 4: This plot shows the average number of steps to goal smoothed over five episodes in the FourRooms domain. Shaded region represents 1 standard error across 100 runs.
  • Figure 5: Obstacles and subgoals for GridBall and PinBall. The larger circles show the initiation set boundaries. Subgoals are defined in position space.
  • ...and 19 more figures

Theorems & Definitions (6)

  • Lemma 1
  • Proposition 2
  • Proposition 3
  • Proposition 4: Convergence rate of tabular value iteration for the biased update
  • Proposition 5
  • Remark 6