Towards Principled Representation Learning from Videos for Reinforcement Learning

TL;DR

The theoretical investigation into principled approaches for representation learning is initiated and it is proved upper bounds for temporal contrastive learning and forward modeling in the presence of only iid noise, which partially explains why reinforcement learning with video pre-training is hard.

Abstract

We study pre-training representations for decision-making using video data, which is abundantly available for tasks such as game agents and software testing. Even though significant empirical advances have been made on this problem, a theoretical understanding remains absent. We initiate the theoretical investigation into principled approaches for representation learning and focus on learning the latent state representations of the underlying MDP using video data. We study two types of settings: one where there is iid noise in the observation, and a more challenging setting where there is also the presence of exogenous noise, which is non-iid noise that is temporally correlated, such as the motion of people or cars in the background. We study three commonly used approaches: autoencoding, temporal contrastive learning, and forward modeling. We prove upper bounds for temporal contrastive learning and forward modeling in the presence of only iid noise. We show that these approaches can learn the latent state and use it to do efficient downstream RL with polynomial sample complexity. When exogenous noise is also present, we establish a lower bound result showing that the sample complexity of learning from video data can be exponentially worse than learning from action-labeled trajectory data. This partially explains why reinforcement learning with video pre-training is hard. We evaluate these representational learning methods in two visual domains, yielding results that are consistent with our theoretical findings.

Figures (14)

• Figure 1: A flowchart of our video pre-training phase. Left: We assume access to a large set of videos (or, unlabeled episodes). Center: A representation learning method is used to train a model $\phi$ which maps an observation to a vector representation. Right: This representation can be used in a downstream task to do reinforcement learning or visualize the latent world state.
• Figure 2: RL experiments in the GridWorld environment.
• Figure 3: Decoded image reconstructions from different latent representation learning methods in the GridWorld environment. We train a decoder on top of frozen representations trained with the three video pre-training approaches. Top row: shows an example from the setting where there is no exogenous noise. Bottom row: shows an example with exogenous noise.
• Figure 4: RL experiments using different latent representations for the ViZDoom environment.
• Figure 5: Decoded image reconstructions from different representation learning methods in the ViZDoom environment. We train a decoder on top of frozen representations trained with the three video pre-training approaches. Here, we show an example where we also have exogenous noise.

Theorems & Definitions (15)

• Theorem 1: Forward Modeling Result
• Theorem 2: Margin Relation
• Theorem 3: Lower Bound for Video
• Lemma 1: Property of Noise-Free policy
• Lemma 2: Distribution over Pairs
• Proposition 4: Generalization Bound
• Proposition 5: Recovering Endogenous State.
• Theorem 6
• Proposition 7: PAC RL Bound
• Theorem 8: Wrapping up the proof.