Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning

TL;DR

This work introduces Multistep Quasimetric Estimation (MQE), an offline goal-conditioned RL method that blends multistep value propagation with a quasimetric distance representation to achieve strong horizon generalization and compositional stitching. MQE defines a quasimetric distance via Metric Residual Networks (MRN) so that $Q_g(s,a) = V_g(g) e^{-d((s,a),g)}$ and $V_g(s) = V_g(g) e^{-d(s,g)}$, with $d((s,a),g) = \,\text{MRN}(\varphi(s,a), \psi(g))$ and $d(s,g) = \,\text{MRN}(\psi(s), \psi(g))$, enabling both local and global value propagation. The framework introduces multistep backups through waypoint sampling $s_t^w$ and two loss terms, $\mathcal{L}_{\mathcal{T}}$ and $\mathcal{L}_{\mathcal{T}_\beta}$ (via a LINEX divergence), along with an action-invariance regularizer $\mathcal{L}_{\mathcal{I}}$ and a policy extraction objective that uses BC-DDPG to encourage goal-reaching actions. Empirically, MQE achieves SoTA performance on long-horizon offline benchmarks (OGBench) and demonstrates real-world compositional generalization on BridgeData with up to 4 sequential tasks, highlighting its practical impact for scalable, horizon-robust GCRL in robotics and complex control tasks. The combination of multistep TD-like propagation with a globally constrained quasimetric distance yields improved horizon generalization, stability in training from noisy offline data, and a simpler, end-to-end pipeline without hierarchical components.

Abstract

Learning how to reach goals in an environment is a longstanding challenge in AI, yet reasoning over long horizons remains a challenge for modern methods. The key question is how to estimate the temporal distance between pairs of observations. While temporal difference methods leverage local updates to provide optimality guarantees, they often perform worse than Monte Carlo methods that perform global updates (e.g., with multi-step returns), which lack such guarantees. We show how these approaches can be integrated into a practical GCRL method that fits a quasimetric distance using a multistep Monte-Carlo return. We show our method outperforms existing GCRL methods on long-horizon simulated tasks with up to 4000 steps, even with visual observations. We also demonstrate that our method can enable stitching in the real-world robotic manipulation domain (Bridge setup). Our approach is the first end-to-end GCRL method that enables multistep stitching in this real-world manipulation domain from an unlabeled offline dataset of visual observations.

Figures (11)

• Figure 1: In this paper, we present Multistep Quasimetric Estimation (). Unlike prior work in quasimetric distance learning that use single-step TD updates wang2023optimal or Monte-Carlo updates myers2024learning, is the first work to incorporate multistep returns with real-world success.
• Figure 2: Tasks from various state and pixel-based environments for OGBench. Antmaze-colossal is 50% larger than any other mazes available on OGBench, and in stitch datasets, test the agent's ability to generalize over horizon that is up to 1000% longer.
• Figure 3: Comparisons of against prior methods on OGBench.
• Figure 4: We evaluate on multi-stage manipulation tasks on BridgeData. Below are examples of the starting observations and goal image being passed in.
• Figure 5: Task progress on BridgeData tasks based on how many consecutive tasks the agent is required to perform, plotted with both the mean and the standard error bars.