Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL

TL;DR

The paper addresses the difficulty of long-horizon offline goal-conditioned RL by reformulating hierarchical decision-making as autoregressive sequence modeling. It introduces CoGHP, a unified framework that autoregressively generates a sequence of latent subgoals $z_{1:H}$ and a primitive action $a$ conditioned on the current state $s$ and goal $g$, using an MLP-Mixer backbone with a learnable causal mixer to enable cross-token communication. A shared value function $V_{\psi}(s,g)$ guides end-to-end training across all sequence elements via advantage-weighted objectives, enabling gradients to propagate through subgoals and actions. Empirically, CoGHP outperforms strong offline baselines on challenging navigation and manipulation benchmarks, and ablations demonstrate the superiority of the MLP-Mixer backbone and the critical role of the causal mixer and teacher forcing. This approach advances long-horizon offline control by enabling multiple intermediate decisions within a single cohesive model, potentially impacting practice in robotics and autonomous systems.

Abstract

Offline goal-conditioned reinforcement learning remains challenging for long-horizon tasks. While hierarchical approaches mitigate this issue by decomposing tasks, most existing methods rely on separate high- and low-level networks and generate only a single intermediate subgoal, making them inadequate for complex tasks that require coordinating multiple intermediate decisions. To address this limitation, we draw inspiration from the chain-of-thought paradigm and propose the Chain-of-Goals Hierarchical Policy (CoGHP), a novel framework that reformulates hierarchical decision-making as autoregressive sequence modeling within a unified architecture. Given a state and a final goal, CoGHP autoregressively generates a sequence of latent subgoals followed by the primitive action, where each latent subgoal acts as a reasoning step that conditions subsequent predictions. To implement this efficiently, we pioneer the use of an MLP-Mixer backbone, which supports cross-token communication and captures structural relationships among state, goal, latent subgoals, and action. Across challenging navigation and manipulation benchmarks, CoGHP consistently outperforms strong offline baselines, demonstrating improved performance on long-horizon tasks.

Figures (9)

• Figure 1: Chain-of-Goals Hierarchical Policy (CoGHP). CoGHP autoregressively generates a sequence of latent subgoals and the primitive action within a unified model. Each subgoal serves as a reasoning token, providing the agent with sufficient guidance to reach the goal. Autoregressive generation ensures that later predictions build upon earlier ones while maintaining awareness of the final goal. To ensure that the subgoal closest to the agent carries the most informative signal, the sequence is generated in reverse order, beginning with the farthest subgoal relative to the current state and progressing toward the nearest one.
• Figure 2: Autoregressive Sequence Generation in CoGHP. The policy autoregressively generates latent subgoals in order from most distant ($z_H$) to nearest ($z_1$) from the current state, and the primitive action $a$. At step $i$, the MLP-Mixer processes state embedding $e_o$, goal embedding $e_g$, previously generated subgoals, and remaining initial tokens to output $z_i$. This sequential generation ensures that each subgoal leverages information from all previously generated waypoints, enabling comprehensive hierarchical reasoning. During training, we apply teacher forcing by providing ground-truth subgoal embeddings to prevent error accumulation.
• Figure 3: Evaluation Environments. Our experiments utilize environments from the OGBench suite. Top row: Navigation capabilities are tested in maze-medium, maze-large, and maze-giant (left to right) with increasing complexity using both Point mass and Ant agents. Bottom row: Manipulation skills are evaluated in cube variants (left and center) involving single to triple cube tasks, and the scene environment (rightmost), which requires multi-step interaction sequences such as unlocking, opening, and manipulating objects.
• Figure 4: Subgoal Visualization. An agent located in the bottom-right corner is tasked with reaching the goal in the top-left. Here, the policy outputs three latent subgoals, plotted as blue, green, and red dots, ordered from nearest to farthest relative to the agent. The full version is in the Appendix \ref{['appendix:subgoal_vis']}.
• Figure 5: CoGHP Architecture. The framework comprises (1) a MLP-Mixer-based hierarchical policy that implements sequence generation for hierarchical control, autoregressively generating latent subgoals $z_H, \ldots, z_1$ ordered from farthest to nearest, and the primitive action $a$, and (2) a shared goal-conditioned value function $V_\psi(s,e_g)$ providing unified training signals for both subgoal generation and action prediction. The MLP-Mixer-based hierarchical policy takes H+3 tokens as input and processes them through alternating token-mixing, causal token-mixing, and channel-mixing layers to generate the output sequence.