Video-Enhanced Offline Reinforcement Learning: A Model-Based Approach

TL;DR

VeoRL tackles offline visual reinforcement learning by exploiting freely available unlabeled videos to broaden an agent's world model through latent control behaviors. It introduces a latent-behavior abstraction (BAN) to discretize high-level actions, and a two-stream world model with a trunk net for real actions and a plan net for latent behaviors, both guiding model-based policy learning with an intrinsic reward that aligns short- and long-horizon dynamics. The approach yields substantial gains across Meta-World, CARLA, and MineDojo, including offline-to-online transfer, by transferring commonsense control policies and physical dynamics from out-of-domain video data. While the results demonstrate strong performance and generalization, the method incurs additional computational overhead due to the dual-branch architecture and video-based training, suggesting future work on efficiency improvements.

Abstract

Offline reinforcement learning (RL) enables policy optimization using static datasets, avoiding the risks and costs of extensive real-world exploration. However, it struggles with suboptimal offline behaviors and inaccurate value estimation due to the lack of environmental interaction. We present Video-Enhanced Offline RL (VeoRL), a model-based method that constructs an interactive world model from diverse, unlabeled video data readily available online. Leveraging model-based behavior guidance, our approach transfers commonsense knowledge of control policy and physical dynamics from natural videos to the RL agent within the target domain. VeoRL achieves substantial performance gains (over 100% in some cases) across visual control tasks in robotic manipulation, autonomous driving, and open-world video games.

Figures (10)

• Figure 1: Overview of the training setup of VeoRL that leads to improved offline RL performance.a, We extract latent behavior abstractions from task-agnostic, unannotated natural video data to enrich the world model's commonsense understanding of the physical world. By interacting with this world model, the agent performs policy optimization guided explicitly by the latent policies learned from the natural videos. b, Overall performance of VeoRL. It demonstrates significant improvements over existing offline RL methods across different visual control benchmarks, including Meta-World robotic manipulation, CARLA autonomous driving, and Minecraft open-world video games. We present the average performance on all tasks on each benchmark.
• Figure 2: Model architecture.a, We construct a discrete, high-level latent action space by training the BAN, enabling forward dynamics modeling independent of real actions. b, The visualization of model-based actor-critic learning at a single rollout step. We leverage $F_\text{BC}$ to replay the video-informed latent behaviors, serving as the inputs of the actor and critic for producing goal-conditioned policies and value estimations, as well as the plan net for generating a long-term state rollout.
• Figure 3: Performance on Meta-World robotic manipulation tasks in episode return. Error bars indicate standard deviation across the $50$ evaluation episodes, with $3$ random training seeds.
• Figure 4: Experiments of autonomous driving.a, Showcases of the source NuScenes and target CARLA datasets. b, Performance comparison on CARLA, measured by averaged episode returns.
• Figure 5: Experiments of the MineDojo 3D navigation and control tasks.a, Showcases of source online videos and target offline datasets. b, Performance comparison in success rate.