Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reconciling Spatial and Temporal Abstractions for Goal Representation

Mehdi Zadem, Sergio Mover, Sao Mai Nguyen

TL;DR

This work tackles the scalability of goal-conditioned hierarchical RL by reconciling spatial and temporal abstractions. It introduces STAR, a three-level Feudal HRL framework that simultaneously learns a reachability-aware spatial abstraction of goals and temporal subgoal planning across multiple time scales. The authors provide regret-like theoretical guarantees showing sub-optimality bounds for policies learned under reachability-aware abstractions and demonstrate improved data efficiency on challenging continuous-control tasks with high-dimensional state spaces. Empirically, STAR outperforms baselines that rely on only spatial or only temporal abstractions, evidencing improved scalability and more sample-efficient learning in Ant Maze, Ant Fall, and Ant Maze Cam environments. The approach holds promise for robust, scalable HRL in complex, continuous domains and can be extended to stochastic and non-Markovian settings in future work.

Abstract

Goal representation affects the performance of Hierarchical Reinforcement Learning (HRL) algorithms by decomposing the complex learning problem into easier subtasks. Recent studies show that representations that preserve temporally abstract environment dynamics are successful in solving difficult problems and provide theoretical guarantees for optimality. These methods however cannot scale to tasks where environment dynamics increase in complexity i.e. the temporally abstract transition relations depend on larger number of variables. On the other hand, other efforts have tried to use spatial abstraction to mitigate the previous issues. Their limitations include scalability to high dimensional environments and dependency on prior knowledge. In this paper, we propose a novel three-layer HRL algorithm that introduces, at different levels of the hierarchy, both a spatial and a temporal goal abstraction. We provide a theoretical study of the regret bounds of the learned policies. We evaluate the approach on complex continuous control tasks, demonstrating the effectiveness of spatial and temporal abstractions learned by this approach. Find open-source code at https://github.com/cosynus-lix/STAR.

Reconciling Spatial and Temporal Abstractions for Goal Representation

TL;DR

This work tackles the scalability of goal-conditioned hierarchical RL by reconciling spatial and temporal abstractions. It introduces STAR, a three-level Feudal HRL framework that simultaneously learns a reachability-aware spatial abstraction of goals and temporal subgoal planning across multiple time scales. The authors provide regret-like theoretical guarantees showing sub-optimality bounds for policies learned under reachability-aware abstractions and demonstrate improved data efficiency on challenging continuous-control tasks with high-dimensional state spaces. Empirically, STAR outperforms baselines that rely on only spatial or only temporal abstractions, evidencing improved scalability and more sample-efficient learning in Ant Maze, Ant Fall, and Ant Maze Cam environments. The approach holds promise for robust, scalable HRL in complex, continuous domains and can be extended to stochastic and non-Markovian settings in future work.

Abstract

Goal representation affects the performance of Hierarchical Reinforcement Learning (HRL) algorithms by decomposing the complex learning problem into easier subtasks. Recent studies show that representations that preserve temporally abstract environment dynamics are successful in solving difficult problems and provide theoretical guarantees for optimality. These methods however cannot scale to tasks where environment dynamics increase in complexity i.e. the temporally abstract transition relations depend on larger number of variables. On the other hand, other efforts have tried to use spatial abstraction to mitigate the previous issues. Their limitations include scalability to high dimensional environments and dependency on prior knowledge. In this paper, we propose a novel three-layer HRL algorithm that introduces, at different levels of the hierarchy, both a spatial and a temporal goal abstraction. We provide a theoretical study of the regret bounds of the learned policies. We evaluate the approach on complex continuous control tasks, demonstrating the effectiveness of spatial and temporal abstractions learned by this approach. Find open-source code at https://github.com/cosynus-lix/STAR.
Paper Structure (30 sections, 3 theorems, 24 equations, 5 figures, 3 tables, 1 algorithm)

This paper contains 30 sections, 3 theorems, 24 equations, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Spatio-Temporal Abstraction via Reachability
  4. A 3-level Hierarchical Algorithm For Temporal Abstraction
  5. Refining $\mathcal{N}$ via Reachability Analysis
  6. Refinement Computation
  7. Theoretical Properties of the Refinement
  8. Experimental Evaluation
  9. Environment Setup
  10. Comparative Analysis
  11. Representation Analysis
  12. Conclusion
  13. Proofs of Theorems
  14. Proof of Theorem 1:
  15. Proof of Lemma 1 and Theorem 2:
  16. ...and 15 more sections

Key Result

Theorem 1

Let $M$ be a deterministic environment with task goal $g^{*} \in \mathcal{S}$ and $r_{ext}(s) = - \lVert g^{*} - s \rVert_2$. Let $\mathcal{N} : \mathcal{S} \rightarrow 2^\mathcal{S}$ be a reachability-aware abstraction with respect to $\mathcal{T}^*_\text{high}$. Then, for $s_0 \in \mathcal{T}^*_\t where $V_\pi(s)$ is the value function for a policy $\pi$Sutton1998. Moreover, if there exists a $B

Figures (5)

  • Figure 1: Architecture of STAR. The algorithm's inputs are the initial state $s_0$, the task goal $g^{*}$, and an initial abstraction $\mathcal{N}_0$. STAR runs in a feedback loop a Feudal HRL algorithm (dashed red block) and an abstraction refinement (blue box). The solid red blocks show the HRL agents (Commander, Tutor, Controller). The agents run at different timescales ($k > l > 1$), shown with the solid, dashed, and dotted lines carrying the feedback from the environment to the agents. The Refinement uses as inputs the past episodes ($\mathcal{D}$) and a the list of abstract goals ($\mathcal{E}$) visited during the last episode, and outputs an abstraction.
  • Figure 2: Ant environments
  • Figure 3: Comparative evaluation averaged over 10 runs for STAR, GARA, HRAC, HIRO and LESSON.
  • Figure 4: Frequency of goals visited by the Commander when evaluating a policy learned after 1M, 2M, and 3M timesteps (averaged over 5 different evaluations with 500 maximum timesteps). The subdivision of the mazes represent (abstract) goals. The color gradient represents the frequency of visits of each goal. Grey areas correspond to the obstacles of the environment in Ant Maze.
  • Figure 5: Frequency of goals visited by the Commander when evaluating a policy learned after 1M, 2M, and 3M timesteps (averaged over 5 different evaluations with 500 maximum timesteps). The subdivision of the mazes represent (abstract) goals. The color gradient represents the frequency of visits of each goal. Grey areas correspond to the platforms of the environment in Ant Fall. The red box is the movable block. Observe that, around 3M timesteps, goals are further split across the $z$-axis above the pit.

Theorems & Definitions (6)

  • Definition 1: Pairwise Reachability Property
  • Definition 2: Pairwise Reachability-Aware Refinement
  • Definition 3: Reachability-Aware Abstraction
  • Theorem 1: Sub-optimal Learning
  • Lemma 1
  • Theorem 2