Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL

Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, Chongjie Zhang

TL;DR

This paper addresses offline goal-conditioned RL with sparse rewards by generalizing goal-conditioned supervised learning (GCSL) into Weighted GCSL (WGCSL). WGCSL introduces a three-part weighting scheme—discounted relabeling weight (DRW), goal-conditioned exponential advantage weight (GEAW), and best-advantage weight (BAW)—and proves that it optimizes an equivalent lower bound to the goal-conditioned RL objective, enabling monotonic policy improvement. The authors provide a theoretical bridge between goal-conditioned RL and weighted supervised learning, and demonstrate strong empirical gains on a new offline benchmark spanning ten manipulation tasks, especially in high-dimensional settings like HandReach and with random datasets. The work offers a simple, stable, and effective offline approach for learning multi-goal policies from self-generated relabeled data, with broad implications for real-world robotics and offline RL applications.

Abstract

Solving goal-conditioned tasks with sparse rewards using self-supervised learning is promising because of its simplicity and stability over current reinforcement learning (RL) algorithms. A recent work, called Goal-Conditioned Supervised Learning (GCSL), provides a new learning framework by iteratively relabeling and imitating self-generated experiences. In this paper, we revisit the theoretical property of GCSL -- optimizing a lower bound of the goal reaching objective, and extend GCSL as a novel offline goal-conditioned RL algorithm. The proposed method is named Weighted GCSL (WGCSL), in which we introduce an advanced compound weight consisting of three parts (1) discounted weight for goal relabeling, (2) goal-conditioned exponential advantage weight, and (3) best-advantage weight. Theoretically, WGCSL is proved to optimize an equivalent lower bound of the goal-conditioned RL objective and generates monotonically improved policies via an iterated scheme. The monotonic property holds for any behavior policies, and therefore WGCSL can be applied to both online and offline settings. To evaluate algorithms in the offline goal-conditioned RL setting, we provide a benchmark including a range of point and simulated robot domains. Experiments in the introduced benchmark demonstrate that WGCSL can consistently outperform GCSL and existing state-of-the-art offline methods in the fully offline goal-conditioned setting.

Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL

TL;DR

This paper addresses offline goal-conditioned RL with sparse rewards by generalizing goal-conditioned supervised learning (GCSL) into Weighted GCSL (WGCSL). WGCSL introduces a three-part weighting scheme—discounted relabeling weight (DRW), goal-conditioned exponential advantage weight (GEAW), and best-advantage weight (BAW)—and proves that it optimizes an equivalent lower bound to the goal-conditioned RL objective, enabling monotonic policy improvement. The authors provide a theoretical bridge between goal-conditioned RL and weighted supervised learning, and demonstrate strong empirical gains on a new offline benchmark spanning ten manipulation tasks, especially in high-dimensional settings like HandReach and with random datasets. The work offers a simple, stable, and effective offline approach for learning multi-goal policies from self-generated relabeled data, with broad implications for real-world robotics and offline RL applications.

Abstract

Solving goal-conditioned tasks with sparse rewards using self-supervised learning is promising because of its simplicity and stability over current reinforcement learning (RL) algorithms. A recent work, called Goal-Conditioned Supervised Learning (GCSL), provides a new learning framework by iteratively relabeling and imitating self-generated experiences. In this paper, we revisit the theoretical property of GCSL -- optimizing a lower bound of the goal reaching objective, and extend GCSL as a novel offline goal-conditioned RL algorithm. The proposed method is named Weighted GCSL (WGCSL), in which we introduce an advanced compound weight consisting of three parts (1) discounted weight for goal relabeling, (2) goal-conditioned exponential advantage weight, and (3) best-advantage weight. Theoretically, WGCSL is proved to optimize an equivalent lower bound of the goal-conditioned RL objective and generates monotonically improved policies via an iterated scheme. The monotonic property holds for any behavior policies, and therefore WGCSL can be applied to both online and offline settings. To evaluate algorithms in the offline goal-conditioned RL setting, we provide a benchmark including a range of point and simulated robot domains. Experiments in the introduced benchmark demonstrate that WGCSL can consistently outperform GCSL and existing state-of-the-art offline methods in the fully offline goal-conditioned setting.
Paper Structure (59 sections, 8 theorems, 38 equations, 16 figures, 4 tables, 1 algorithm)

This paper contains 59 sections, 8 theorems, 38 equations, 16 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Markov Decision Process and offline RL
  4. Goal-conditioned RL
  5. Goal-conditioned Supervised Learning
  6. Weighted Goal-Conditioned Supervised Learning
  7. Revisiting Goal-conditioned Supervised Learning
  8. A Motivating Example
  9. Connection between Goal-Conditioned RL and SL
  10. A More General Weighting Scheme
  11. Policy Improvement via WGCSL
  12. Algorithm
  13. Experiments
  14. Environments and Experimental Settings
  15. Experimental Results
  16. ...and 44 more sections

Key Result

Theorem 1

Assume a finite-horizon discrete MDP, a stochastic discrete policy $\pi$ which selects actions with non-zero probability and a sparse reward function $r(s_t,a_t,g)=1[\phi(s_t)=g]$, where $\phi$ is the state-to-goal mapping and $1[\phi(s_t)=g]$ is an indicator function. Given trajectories $\tau=(s_1, where $J_{surr}(\pi)=\frac{1}{T}\mathbb{E}_{g\sim p(g), \tau\sim\pi_{b}(\cdot|g)} \left[ \sum_{t=1}

Figures (16)

  • Figure 1: Diagram of WGCSL in the offline goal-conditioned RL setting. WGCSL samples data from the offline dataset and relabels them with hindsight goals. After that, WGCSL learns a value function and updates the policy via a weighted supervised scheme.
  • Figure 2: Visualization of the trajectories generated by GCSL (blue) and WGCSL (orange) in the PointReach task.
  • Figure 3: Goal-conditioned tasks: (a) PointReach, (b) PointRooms, (c) Reacher, (d) SawyerReach, (e) SawyerDoor, (f) FetchReach, (g) FetchPush, (h) FetchSlide, (i) FetchPick, (j) HandReach.
  • Figure 4: Performance on expert (top row) and random (bottom row) offline dataset of four simulated manipulation tasks. Results are averaged over 5 random seeds and the shaded region represents the standard deviation.
  • Figure 5: Ablation studies of WGCSL in the offline setting.
  • ...and 11 more figures

Theorems & Definitions (8)

  • Theorem 1
  • Corollary 1
  • Proposition 1
  • Proposition 2
  • Theorem
  • Corollary
  • Proposition
  • Proposition