Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning from Passive Data via Latent Intentions

Dibya Ghosh, Chethan Bhateja, Sergey Levine

TL;DR

The paper addresses enabling reinforcement learning from abundant passive data by introducing intention-conditioned value functions (ICVFs) that model how likely future outcomes are when following latent intentions $z$. It develops a TD-based, multilinear representation $\hat{V}_\theta(s, s_+, z)=\phi_\theta(s)^T T_\theta(z) \psi_\theta(s_+)$, enabling joint learning of state, outcome, and intention representations from observations without rewards or actions. Empirically, ICVF pretraining improves downstream RL performance and value-function fidelity across diverse domains, including cross-embodiment video data (XMagical), D4RL Antmaze, and YouTube Atari, often approaching oracle performance and demonstrating robustness to embodiment gaps. This approach offers a scalable pathway to extract control-relevant features from passive data, with potential impact on data efficiency and generalization in real-world RL applications.

Abstract

Passive observational data, such as human videos, is abundant and rich in information, yet remains largely untapped by current RL methods. Perhaps surprisingly, we show that passive data, despite not having reward or action labels, can still be used to learn features that accelerate downstream RL. Our approach learns from passive data by modeling intentions: measuring how the likelihood of future outcomes change when the agent acts to achieve a particular task. We propose a temporal difference learning objective to learn about intentions, resulting in an algorithm similar to conventional RL, but which learns entirely from passive data. When optimizing this objective, our agent simultaneously learns representations of states, of policies, and of possible outcomes in an environment, all from raw observational data. Both theoretically and empirically, this scheme learns features amenable for value prediction for downstream tasks, and our experiments demonstrate the ability to learn from many forms of passive data, including cross-embodiment video data and YouTube videos.

Reinforcement Learning from Passive Data via Latent Intentions

TL;DR

The paper addresses enabling reinforcement learning from abundant passive data by introducing intention-conditioned value functions (ICVFs) that model how likely future outcomes are when following latent intentions . It develops a TD-based, multilinear representation , enabling joint learning of state, outcome, and intention representations from observations without rewards or actions. Empirically, ICVF pretraining improves downstream RL performance and value-function fidelity across diverse domains, including cross-embodiment video data (XMagical), D4RL Antmaze, and YouTube Atari, often approaching oracle performance and demonstrating robustness to embodiment gaps. This approach offers a scalable pathway to extract control-relevant features from passive data, with potential impact on data efficiency and generalization in real-world RL applications.

Abstract

Passive observational data, such as human videos, is abundant and rich in information, yet remains largely untapped by current RL methods. Perhaps surprisingly, we show that passive data, despite not having reward or action labels, can still be used to learn features that accelerate downstream RL. Our approach learns from passive data by modeling intentions: measuring how the likelihood of future outcomes change when the agent acts to achieve a particular task. We propose a temporal difference learning objective to learn about intentions, resulting in an algorithm similar to conventional RL, but which learns entirely from passive data. When optimizing this objective, our agent simultaneously learns representations of states, of policies, and of possible outcomes in an environment, all from raw observational data. Both theoretically and empirically, this scheme learns features amenable for value prediction for downstream tasks, and our experiments demonstrate the ability to learn from many forms of passive data, including cross-embodiment video data and YouTube videos.
Paper Structure (25 sections, 2 theorems, 14 equations, 16 figures, 4 tables)

This paper contains 25 sections, 2 theorems, 14 equations, 16 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Reinforcement Learning using Passive Data
  4. Learning Representations from ICVFs
  5. Learning ICVFs from Passive Data
  6. Related Work
  7. Experiments
  8. Experimental setup and comparisons
  9. Learning from the same embodiment
  10. Learning from different embodiments
  11. Atari 2600: Learning from YouTube videos
  12. Conclusion
  13. Didactic Example of ICVF
  14. Proofs of Statements
  15. Experimental Setup
  16. ...and 10 more sections

Key Result

Proposition 1

Suppose $\phi, \psi, T$ form an approximation to the true ICVF with error $\epsilon$, that is $\forall z \in {\mathcal{Z}}$, For all rewards $r(s)$ and intentions $z \in {\mathcal{Z}}$, $\exists \theta_r^z \in \mathbb{R}^d$ s.t.

Figures (16)

  • Figure 1: D4RL Antmaze tasks: We compare ICVF representations pre-trained on passive data from antmaze-large-diverse-v2 to representations from other methods for downstream RL performance (left) and for fidelity of value approximations (middle). Ablating our approach, we find both components of our ICVF method to be important for performance: learning future outcomes from many intents, and the use of the multi-linear decomposition (middle). (Additional Antmaze tasks in Table \ref{['tab:antmaze']} and \ref{['appendix:d4rl_extra']}.
  • Figure 2: XMagical: (top) A visualization of the different agent embodiments. (bottom) Downstream performance on the Gripper embodiment after pretraining from passive same-embodiment or cross-embodiment data. ICVF representations lead to the highest downstream performance amongst all comparisons.
  • Figure 3: YouTube videos of Atari games include corruptions such as color and angle shifts, lighting differences, and text overlays.
  • Figure 4: Atari 2600 with YouTube Videos: Final performance of QRDQN-CQL agent initialized with representations learned from our YouTube video dataset. In three of four games, ICVF representations lead to improved performance by large margins.
  • Figure 5: ICVF representations on YouTube video are better able to predict actions of an QRDQN agent trained with $100\times$ more data, as measured by a linear probe. Value probe in Appendix \ref{['appendix:atari_extra']}
  • ...and 11 more figures

Theorems & Definitions (5)

  • Remark 1
  • Proposition 1: Downstream value approximation
  • Remark 2
  • Proposition 2: Downstream value approximation
  • proof