LuciBot: Automated Robot Policy Learning from Generated Videos

TL;DR

This work tackles scalable supervision for embodied manipulation by leveraging a general-purpose video generation model to imagine task demonstrations from an initial frame and a textual description. It extracts rich multimodal supervision signals, including $6$-D poses, $2$-D segmentation masks, and depth, as well as contact and affordance cues, and uses these to guide trajectory optimization via CMA-ES in simulation. The approach is evaluated on 10 tasks spanning deformable, articulated, and rigid materials in the Genesis simulator, where LuciBot outperforms baselines that rely on LLM/VLM rewards or domain-tuned video models. Real-world transfer is demonstrated through digital twins, showing practical potential for scalable, generalizable supervision without task-specific fine-tuning, albeit with limitations on long-horizon video generation.

Abstract

Automatically generating training supervision for embodied tasks is crucial, as manual designing is tedious and not scalable. While prior works use large language models (LLMs) or vision-language models (VLMs) to generate rewards, these approaches are largely limited to simple tasks with well-defined rewards, such as pick-and-place. This limitation arises because LLMs struggle to interpret complex scenes compressed into text or code due to their restricted input modality, while VLM-based rewards, though better at visual perception, remain limited by their less expressive output modality. To address these challenges, we leverage the imagination capability of general-purpose video generation models. Given an initial simulation frame and a textual task description, the video generation model produces a video demonstrating task completion with correct semantics. We then extract rich supervisory signals from the generated video, including 6D object pose sequences, 2D segmentations, and estimated depth, to facilitate task learning in simulation. Our approach significantly improves supervision quality for complex embodied tasks, enabling large-scale training in simulators.

Figures (11)

• Figure 1: We present LuciBot, a framework that utilizes a general-purpose video generation model to obtain rich supervision signals. Leveraging the semantic guidance of video generation model, LuciBot extract rich supervision to facilitate complicated embodied AI tasks.
• Figure 2: Our pipeline consists of three stages: (1) Generating and selecting a valid video, (2) Extracting various training supervision from the selected video, and (3) Leveraging this information for policy learning.
• Figure 3: We propose 10 challenging tasks involving diverse materials, including liquid, elastic, elasto-plastic, granular, articulated, and rigid objects. Designing reward functions for those deformable tasks is inherently difficult, even with manual efforts.
• Figure 4: We display the end frames executing our policy, demonstrating that our method can reasonably perform these tasks.
• Figure 5: We show the best qualitative results for all baselines of multiple trials.