Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transitive RL: Value Learning via Divide and Conquer

Seohong Park, Aditya Oberai, Pranav Atreya, Sergey Levine

TL;DR

Transitive RL (TRL) addresses the horizon-bias challenge in offline goal-conditioned RL by using a divide-and-conquer value-learning scheme that leverages the triangle inequality to update values via subgoals, potentially reducing Bellman recursions to $O(\log T)$. It replaces the hard subgoal max with soft expectile backups and confines subgoals to in-trajectory (behavioral) states, augmented with distance-based re-weighting for stability. Empirically, TRL achieves state-of-the-art or strong results on long-horizon robotic tasks and competitive performance on standard benchmarks, without task-specific horizon tuning. Overall, the work demonstrates that a divide-and-conquer, triangle-inequality-guided approach can mitigate the curse of horizon in offline GCRL and offer a scalable alternative to TD and MC methods.

Abstract

In this work, we present Transitive Reinforcement Learning (TRL), a new value learning algorithm based on a divide-and-conquer paradigm. TRL is designed for offline goal-conditioned reinforcement learning (GCRL) problems, where the aim is to find a policy that can reach any state from any other state in the smallest number of steps. TRL converts a triangle inequality structure present in GCRL into a practical divide-and-conquer value update rule. This has several advantages compared to alternative value learning paradigms. Compared to temporal difference (TD) methods, TRL suffers less from bias accumulation, as in principle it only requires $O(\log T)$ recursions (as opposed to $O(T)$ in TD learning) to handle a length-$T$ trajectory. Unlike Monte Carlo methods, TRL suffers less from high variance as it performs dynamic programming. Experimentally, we show that TRL achieves the best performance in highly challenging, long-horizon benchmark tasks compared to previous offline GCRL algorithms.

Transitive RL: Value Learning via Divide and Conquer

TL;DR

Transitive RL (TRL) addresses the horizon-bias challenge in offline goal-conditioned RL by using a divide-and-conquer value-learning scheme that leverages the triangle inequality to update values via subgoals, potentially reducing Bellman recursions to . It replaces the hard subgoal max with soft expectile backups and confines subgoals to in-trajectory (behavioral) states, augmented with distance-based re-weighting for stability. Empirically, TRL achieves state-of-the-art or strong results on long-horizon robotic tasks and competitive performance on standard benchmarks, without task-specific horizon tuning. Overall, the work demonstrates that a divide-and-conquer, triangle-inequality-guided approach can mitigate the curse of horizon in offline GCRL and offer a scalable alternative to TD and MC methods.

Abstract

In this work, we present Transitive Reinforcement Learning (TRL), a new value learning algorithm based on a divide-and-conquer paradigm. TRL is designed for offline goal-conditioned reinforcement learning (GCRL) problems, where the aim is to find a policy that can reach any state from any other state in the smallest number of steps. TRL converts a triangle inequality structure present in GCRL into a practical divide-and-conquer value update rule. This has several advantages compared to alternative value learning paradigms. Compared to temporal difference (TD) methods, TRL suffers less from bias accumulation, as in principle it only requires recursions (as opposed to in TD learning) to handle a length- trajectory. Unlike Monte Carlo methods, TRL suffers less from high variance as it performs dynamic programming. Experimentally, we show that TRL achieves the best performance in highly challenging, long-horizon benchmark tasks compared to previous offline GCRL algorithms.
Paper Structure (21 sections, 2 theorems, 19 equations, 5 figures, 5 tables, 1 algorithm)

This paper contains 21 sections, 2 theorems, 19 equations, 5 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Previous Approaches to Offline GCRL
  5. Transitive RL
  6. The Challenge
  7. The Idea
  8. Practical Recipe
  9. Value Learning
  10. Policy Extraction
  11. Experiments
  12. Is divide and conquer better than TD and MC on long-horizon tasks?
  13. How does TRL compare to prior methods on standard benchmarks?
  14. Ablation Studies
  15. What's Next?
  16. ...and 6 more sections

Key Result

Lemma A.1

Let $n \geq 2$ be an integer and $C(n) = \frac{1}{n-1} \sum_{k=1}^{n-1} \max(k, n-k)$. Then $C(n) \leq 3n/4$.

Figures (5)

  • Figure 1: Transitive RL. Transitive RL is based on the divide-and-conquer paradigm, which can in theory reduce the number of Bellman recursions to $O(\log T)$ in the best case, unlike TD-based methods.
  • Figure 2: Long-horizon tasks.
  • Figure 3: TRL matches the best, individually tuned TD-$\bm{n}$ baseline, without needing to set $\bm{n}$.
  • Figure 4: TRL achieves strong performance on standard OGBench tasks. While TRL is not specifically designed for short-horizon tasks, it outperforms or matches previous GCRL methods on a standard benchmark.
  • Figure 5: Ablation studies. We ablate three components of TRL and study how they affect performance.

Theorems & Definitions (4)

  • Lemma A.1
  • proof
  • Theorem A.2
  • proof