Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Horizon Generalization in Reinforcement Learning

Vivek Myers, Catherine Ji, Benjamin Eysenbach

TL;DR

The paper addresses horizon generalization in goal-conditioned RL, proposing that policies invariant to planning can generalize from short-horizon to long-horizon goals. By formalizing planning invariance around a quasimetric distance $d(s,g)$ and a planning operator ${\textsc{Plan}}$, it proves that quasimetric policies can be invariant to planning and thus generalize across horizons. It introduces successor-based quasimetric distances $d_{SD}$, develops theoretical results linking planning invariance to horizon generalization, and validates these ideas empirically in high-dimensional and complex environments, observing correlations with reduced Bellman errors. The findings suggest practical directions—especially quasimetric architectures and planning-inspired objectives—for designing RL systems capable of long-horizon reasoning without relying on external planners, with potential impact on robotics and other long-horizon control tasks.

Abstract

We study goal-conditioned RL through the lens of generalization, but not in the traditional sense of random augmentations and domain randomization. Rather, we aim to learn goal-directed policies that generalize with respect to the horizon: after training to reach nearby goals (which are easy to learn), these policies should succeed in reaching distant goals (which are quite challenging to learn). In the same way that invariance is closely linked with generalization is other areas of machine learning (e.g., normalization layers make a network invariant to scale, and therefore generalize to inputs of varying scales), we show that this notion of horizon generalization is closely linked with invariance to planning: a policy navigating towards a goal will select the same actions as if it were navigating to a waypoint en route to that goal. Thus, such a policy trained to reach nearby goals should succeed at reaching arbitrarily-distant goals. Our theoretical analysis proves that both horizon generalization and planning invariance are possible, under some assumptions. We present new experimental results and recall findings from prior work in support of our theoretical results. Taken together, our results open the door to studying how techniques for invariance and generalization developed in other areas of machine learning might be adapted to achieve this alluring property.

Horizon Generalization in Reinforcement Learning

TL;DR

The paper addresses horizon generalization in goal-conditioned RL, proposing that policies invariant to planning can generalize from short-horizon to long-horizon goals. By formalizing planning invariance around a quasimetric distance and a planning operator , it proves that quasimetric policies can be invariant to planning and thus generalize across horizons. It introduces successor-based quasimetric distances , develops theoretical results linking planning invariance to horizon generalization, and validates these ideas empirically in high-dimensional and complex environments, observing correlations with reduced Bellman errors. The findings suggest practical directions—especially quasimetric architectures and planning-inspired objectives—for designing RL systems capable of long-horizon reasoning without relying on external planners, with potential impact on robotics and other long-horizon control tasks.

Abstract

We study goal-conditioned RL through the lens of generalization, but not in the traditional sense of random augmentations and domain randomization. Rather, we aim to learn goal-directed policies that generalize with respect to the horizon: after training to reach nearby goals (which are easy to learn), these policies should succeed in reaching distant goals (which are quite challenging to learn). In the same way that invariance is closely linked with generalization is other areas of machine learning (e.g., normalization layers make a network invariant to scale, and therefore generalize to inputs of varying scales), we show that this notion of horizon generalization is closely linked with invariance to planning: a policy navigating towards a goal will select the same actions as if it were navigating to a waypoint en route to that goal. Thus, such a policy trained to reach nearby goals should succeed at reaching arbitrarily-distant goals. Our theoretical analysis proves that both horizon generalization and planning invariance are possible, under some assumptions. We present new experimental results and recall findings from prior work in support of our theoretical results. Taken together, our results open the door to studying how techniques for invariance and generalization developed in other areas of machine learning might be adapted to achieve this alluring property.
Paper Structure (52 sections, 4 theorems, 40 equations, 11 figures, 1 table)

This paper contains 52 sections, 4 theorems, 40 equations, 11 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning to Reach Goals.
  4. Generalization in RL.
  5. State Abstractions for Decision-Making.
  6. Planning Invariance and Horizon Generalization
  7. Intuition
  8. Preliminaries
  9. Quasimetrics on states.
  10. Successor distances (a quasimetric).
  11. Introducing and Analyzing Horizon Generalization
  12. Definitions of Planning Invariance and Horizon Generalization
  13. Quasimetric Policies are (Nontrivially) Planning Invariant
  14. Quasimetric Policies Generalize over the Horizon
  15. Limitations and Assumptions
  16. ...and 37 more sections

Key Result

theorem 1

Given a deterministic MDP with states ${\mathcal{S}}$, actions ${\mathcal{A}}$, and goal-conditioned Kronecker delta reward function $r_{g}(s) = \delta_{(s,g)}$, define quasimetric policy $\pi_{d}(a\mid s,g)$ and quasimetric planner class $\planclass_{d}$. Then, for every quasimetric planner $\texts

Figures (11)

  • Figure 1: Horizon generalization. A policy generalizes over the horizon if performance for start-goal pairs $(s,g)$ separated by a small temporal distance $d(s,g) < c$ yields improved performance over more distant start-goal pairs $(s',g')$ with $d(s',g') > c$.
  • Figure 2: Visualizing planning invariance. Planning invariance (\ref{['def:planning-invariance']}) means that a policy should take similar actions when directed towards a goal (purple arrow and purple star) as when directed towards an intermediate waypoint (brown arrow and brown star). We visualize a policy with (Right) and without (Left) this property via the misalignment and alignment of actions towards the waypoint and the goal, where the optimal path is tan and the suboptimal path is gray.
  • Figure 3: Invariance to planning leads to horizon generalization.(Left) Invariance to planning maps $(s_{0},\{s_{1}, s_{2}, s_{4}\})$ together in latent space, which results in a shared optimal action. (Right) After successfully reaching the closest waypoint $s_{1}$ in $1$ step, the next optimal action is also shared, meaning any trajectory of length $2$ is optimal. We can repeat this argument for trajectories of length $4,8,\ldots$ until the entire reachable state space is covered.
  • Figure 4: Approximate horizon generalization is still useful.$\textsc{Success}$ when there is horizon generalization. When the success attenuation factor $\eta \geq 0.5$, the $\textsc{Reach}$ goes to $\infty$. For a policy with no horizon generalization ($\eta = 0$), its $\textsc{Reach} = 1$.
  • Figure 5: Quantifying horizon generalization and invariance to planning. On a simple navigation task, we collect short trajectories and train two goal-conditioned policies, comparing both to a random policy. (Top Left) We evaluate on $(s, g)$ pairs of varying distances, observing that metric regression with a quasimetric exhibits strong horizon generalization. (Top Right) In line with our analysis, the policy that has strong horizon generalization is also more invariant to planning: combining that policy with planning does not increase performance. (Bottom Row) We repeat these experiments using function approximation (instead of a tabular model), observing similar trends.
  • ...and 6 more figures

Theorems & Definitions (19)

  • definition 1: Path relaxation operator
  • definition 2: Quasimetric policy
  • definition 3: Planning invariance
  • definition 4: Horizon generalization
  • theorem 1: Quasimetric policies are invariant under $\textsc{Plan}_{d}$
  • theorem 2: Horizon generalization exists
  • remark 1: Horizon generalization is nontrivial
  • definition 5: Path relaxation operator with actions
  • proof
  • definition 6: Quasimetric over distributions
  • ...and 9 more