Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning

TL;DR

The paper tackles long-horizon offline goal-conditioned RL where hierarchical methods like HIQL struggle due to noisy high-level value estimates. It introduces Option-aware Temporally Abstracted (OTA) value learning, which uses temporally extended options to compress the planning horizon and align high-level value signals with the true temporal distance to goals. The OTA loss updates the high-level value via an option-aware TD target, reducing sensitivity to horizon length and noise, and decoupling discounting from the abstraction level. Empirical results on the OGBench suite show OTA markedly improves high-level learning and overall task success, particularly in long-horizon maze and visual robotic manipulation tasks, outperforming baselines including HIQL. The work demonstrates a simple, scalable approach to stabilize and improve long-horizon offline GCRL by leveraging temporal abstraction through the option framework.

Abstract

Offline goal-conditioned reinforcement learning (GCRL) offers a practical learning paradigm in which goal-reaching policies are trained from abundant state-action trajectory datasets without additional environment interaction. However, offline GCRL still struggles with long-horizon tasks, even with recent advances that employ hierarchical policy structures, such as HIQL. Identifying the root cause of this challenge, we observe the following insight. Firstly, performance bottlenecks mainly stem from the high-level policy's inability to generate appropriate subgoals. Secondly, when learning the high-level policy in the long-horizon regime, the sign of the advantage estimate frequently becomes incorrect. Thus, we argue that improving the value function to produce a clear advantage estimate for learning the high-level policy is essential. In this paper, we propose a simple yet effective solution: Option-aware Temporally Abstracted value learning, dubbed OTA, which incorporates temporal abstraction into the temporal-difference learning process. By modifying the value update to be option-aware, our approach contracts the effective horizon length, enabling better advantage estimates even in long-horizon regimes. We experimentally show that the high-level policy learned using the OTA value function achieves strong performance on complex tasks from OGBench, a recently proposed offline GCRL benchmark, including maze navigation and visual robotic manipulation environments.

Figures (9)

• Figure 1: High-level policy is problematic. We evaluate HIQL by varying only the high-level policy while keeping the low-level policy fixed. The x-axis denotes different tasks under maze sizes and data types (See Section \ref{['env_setup']} for task details). Using learned high-level policy (HIQL, $\pi = \pi^\ell \circ \pi^h$), performance drops, whereas using the oracle high-level policy ($\text{HIQL}^\text{OS}$, $\pi = \pi^\ell \circ \pi^h_{\text{oracle}}$) achieves high success rates, indicating the high-level policy is the main bottleneck.
• Figure 2: Value order inconsistency in long-horizon settings. (Left) We collect optimal trajectories from the initial state () to the goal (). (Middle) At each state along the trajectory, we compare the high-level value from HIQL ($\textcolor{blue}{V^h}$) and the optimal ($V^\star$). (Right) To better illustrate value order consistency, we convert the values into temporal distances: HIQL ($\textcolor{red}{d^h}$) and the optimal ($d^\star$).
• Figure 3: Option-aware temporal abstraction. (Left) OTA achieves temporal abstraction by computing the reward and target value from the state reached after executing the option ( i.e., $s^\Omega$). (Right) By leveraging temporal abstraction, OTA provides clear high-level advantage estimates, particularly in long-horizon tasks.
• Figure 4: Evaluation on OGBench. We run $8$ seeds for each dataset and use the performance reported in OGBench for the baselines. For maze tasks, we report the average success rate grouped by maze size. For visual robotic manipulation, we report the average success rate across the four tasks.
• Figure 5: Value and temporal distance estimation. We visualize min-max normalized $V^h, V^h_{\text{OTA}},d^h$, and $d^h_{\text{OTA}}$, and the order consistency ratios $r^c(V^h)$ and $r^c(V^h_{\text{OTA}})$, across six different datasets.

Theorems & Definitions (1)

• Definition 4.1