Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Goal-Conditioned Reinforcement Learning: Problems and Solutions

Minghuan Liu, Menghui Zhu, Weinan Zhang

TL;DR

This survey analyzes goal-conditioned reinforcement learning (GCRL), where policies depend on explicit goals and the environment, introducing goal-augmented MDPs (GA-MDPs) to address generalization and sample-efficiency challenges under sparse rewards. It reviews goal representations (vectors, images, language, and other forms), and organizes solutions into optimization-based methods (e.g., UVFA, reward shaping, self-imitation, planning), sub-goal generation, and relabeling strategies (e.g., HER, curriculum relabeling, foresight). The paper synthesizes techniques for learning across multiple goals, generating intermediate objectives, and reusing data through relabeling to boost data efficiency and exploration. It also outlines future directions, including fully intrinsic skill learning, leveraging offline datasets, and developing large pre-trained goal-conditioned policies, to broaden GCRL applicability and scalability.

Abstract

Goal-conditioned reinforcement learning (GCRL), related to a set of complex RL problems, trains an agent to achieve different goals under particular scenarios. Compared to the standard RL solutions that learn a policy solely depending on the states or observations, GCRL additionally requires the agent to make decisions according to different goals. In this survey, we provide a comprehensive overview of the challenges and algorithms for GCRL. Firstly, we answer what the basic problems are studied in this field. Then, we explain how goals are represented and present how existing solutions are designed from different points of view. Finally, we make the conclusion and discuss potential future prospects that recent researches focus on.

Goal-Conditioned Reinforcement Learning: Problems and Solutions

TL;DR

This survey analyzes goal-conditioned reinforcement learning (GCRL), where policies depend on explicit goals and the environment, introducing goal-augmented MDPs (GA-MDPs) to address generalization and sample-efficiency challenges under sparse rewards. It reviews goal representations (vectors, images, language, and other forms), and organizes solutions into optimization-based methods (e.g., UVFA, reward shaping, self-imitation, planning), sub-goal generation, and relabeling strategies (e.g., HER, curriculum relabeling, foresight). The paper synthesizes techniques for learning across multiple goals, generating intermediate objectives, and reusing data through relabeling to boost data efficiency and exploration. It also outlines future directions, including fully intrinsic skill learning, leveraging offline datasets, and developing large pre-trained goal-conditioned policies, to broaden GCRL applicability and scalability.

Abstract

Goal-conditioned reinforcement learning (GCRL), related to a set of complex RL problems, trains an agent to achieve different goals under particular scenarios. Compared to the standard RL solutions that learn a policy solely depending on the states or observations, GCRL additionally requires the agent to make decisions according to different goals. In this survey, we provide a comprehensive overview of the challenges and algorithms for GCRL. Firstly, we answer what the basic problems are studied in this field. Then, we explain how goals are represented and present how existing solutions are designed from different points of view. Finally, we make the conclusion and discuss potential future prospects that recent researches focus on.
Paper Structure (35 sections, 12 equations, 3 figures, 2 tables)

This paper contains 35 sections, 12 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Problems and Challenges
  3. Standard RL Problems as MDPs
  4. Goal-Augmented MDPs
  5. Basic Challenges
  6. Generalization: Learning over Multiple Tasks
  7. Sample Efficiency: Towards Sparse Rewards
  8. What are Goals?
  9. Desired Property and Feature Vectors
  10. Image Goals
  11. Language Goals
  12. Other Forms of Goals
  13. Expected return as goals.
  14. Upside-down RL: commands as goals.
  15. Algorithms to GCRL
  16. ...and 20 more sections

Figures (3)

  • Figure 1: The mission of policy in complex RL problems. a) Learn to achieve multiple tasks with one single policy; b) Decompose the long-term, hard-reaching goals into easily obtained sub-goals.
  • Figure 2: Typical representations of goals in GCRL: vectors, images and languages.
  • Figure 3: Typical phases in a GCRL algorithm. The agent is provided (or selects by itself) a behavior goal as the target, then interacts with the environments and collects experiences. Before optimizing the policy, the historical data can be relabeled by changing the desired goals.

Theorems & Definitions (3)

  • Definition 1: Desired Goal
  • Definition 2: Achieved Goal
  • Definition 3: Behavioral Goal