Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RoboVIP: Multi-View Video Generation with Visual Identity Prompting Augments Robot Manipulation

Boyang Wang, Haoran Zhang, Shujie Zhang, Jinkun Hao, Mingda Jia, Qi Lv, Yucheng Mao, Zhaoyang Lyu, Jia Zeng, Xudong Xu, Jiangmiao Pang

TL;DR

RoboVIP tackles the data bottleneck in robotic manipulation by enabling scalable, multi-view, temporally coherent data augmentation through a diffusion-based video model conditioned on text and rich visual identities. It integrates an action-informed segmentation pipeline, a million-scale visual identity pool, and a multi-view inpainting diffusion model with frame-wise identity conditioning to produce diverse, consistent observations for vision-language-action and visuomotor policy training. Across simulation and real-robot experiments, RoboVIP yields measurable gains in task success, temporal coherence, and robustness to distractors, outperforming state-of-the-art single-view or identity-agnostic baselines. The approach facilitates plug-and-play data augmentation that can substantially improve policy learning in varied environments, reducing reliance on costly real-world data collection while maintaining high fidelity visuals and cross-view consistency.

Abstract

The diversity, quantity, and quality of manipulation data are critical for training effective robot policies. However, due to hardware and physical setup constraints, collecting large-scale real-world manipulation data remains difficult to scale across diverse environments. Recent work uses text-prompt conditioned image diffusion models to augment manipulation data by altering the backgrounds and tabletop objects in the visual observations. However, these approaches often overlook the practical need for multi-view and temporally coherent observations required by state-of-the-art policy models. Further, text prompts alone cannot reliably specify the scene setup. To provide the diffusion model with explicit visual guidance, we introduce visual identity prompting, which supplies exemplar images as conditioning inputs to guide the generation of the desired scene setup. To this end, we also build a scalable pipeline to curate a visual identity pool from large robotics datasets. Using our augmented manipulation data to train downstream vision-language-action and visuomotor policy models yields consistent performance gains in both simulation and real-robot settings.

RoboVIP: Multi-View Video Generation with Visual Identity Prompting Augments Robot Manipulation

TL;DR

RoboVIP tackles the data bottleneck in robotic manipulation by enabling scalable, multi-view, temporally coherent data augmentation through a diffusion-based video model conditioned on text and rich visual identities. It integrates an action-informed segmentation pipeline, a million-scale visual identity pool, and a multi-view inpainting diffusion model with frame-wise identity conditioning to produce diverse, consistent observations for vision-language-action and visuomotor policy training. Across simulation and real-robot experiments, RoboVIP yields measurable gains in task success, temporal coherence, and robustness to distractors, outperforming state-of-the-art single-view or identity-agnostic baselines. The approach facilitates plug-and-play data augmentation that can substantially improve policy learning in varied environments, reducing reliance on costly real-world data collection while maintaining high fidelity visuals and cross-view consistency.

Abstract

The diversity, quantity, and quality of manipulation data are critical for training effective robot policies. However, due to hardware and physical setup constraints, collecting large-scale real-world manipulation data remains difficult to scale across diverse environments. Recent work uses text-prompt conditioned image diffusion models to augment manipulation data by altering the backgrounds and tabletop objects in the visual observations. However, these approaches often overlook the practical need for multi-view and temporally coherent observations required by state-of-the-art policy models. Further, text prompts alone cannot reliably specify the scene setup. To provide the diffusion model with explicit visual guidance, we introduce visual identity prompting, which supplies exemplar images as conditioning inputs to guide the generation of the desired scene setup. To this end, we also build a scalable pipeline to curate a visual identity pool from large robotics datasets. Using our augmented manipulation data to train downstream vision-language-action and visuomotor policy models yields consistent performance gains in both simulation and real-robot settings.
Paper Structure (27 sections, 12 figures, 3 tables)

This paper contains 27 sections, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Conditioned Video Generation
  4. Visual Augmentation on Robotics
  5. Manipulation Models
  6. Method
  7. Problem Formulation
  8. Action-guided Segmentation of Robots and Interacted Objects
  9. Multi-View Inpainting Video Diffusion Model
  10. Visual Identity Prompting
  11. Experiment
  12. Video Diffusion Model Implementation Details
  13. Video Generation Results
  14. Simulation Results
  15. Real-World Robot Results
  16. ...and 12 more sections

Figures (12)

  • Figure 1: Overview of our RoboVIP Workflow. (1) We extract observation videos from robotics manipulation data with corresponding action data to segment the robot arm and interacted objects for inpainting-based augmentation. (2) A large-scale pool of visual identity prompts is curated from robotics datasets and used as conditioning inputs for our multi-view video diffusion model to conduct diverse visual augmentation. (3) The augmented videos, paired with action information from original robotics manipulation data, are utilized for downstream VLA and visuomotor policy training.
  • Figure 2: Segmentation Pipeline. Our segmentation pipeline comprises two parallel streams: one for robot-arm segmentation and one for interacted-object segmentation. We first use the gripper-action signal to identify accurate keyframe ranges, which is helpful to locate the interacted objects that are not visible in the first or last frame. We then leverage off-the-shelf models such as Cosmos-Reason1 azzolini2025cosmos and SAM2 ravi2024sam, together with several heuristic refinements, to obtain accurate masks in a fully plug-and-play manner.
  • Figure 3: Video Diffusion Model Architecture. Our video diffusion model is conditioned on the segmented multi-view video sequence, structured text prompt, and visual identity prompting to achieve consistent visual augmentation.
  • Figure 4: Visual Identity Curation and Processing Pipeline. Our visual identity is curated by panoptic segmentation from the large-scale robotics dataset ebert2021bridgewalke2023bridgedatakhazatsky2024droid, followed by several scoring criteria filters. In augmentation, we randomly select some from the pool and pack them into one image frame to serve as conditioning for our video diffusion model.
  • Figure 5: Qualitative comparisons of different models on Droid khazatsky2024droid. Our method produces temporally consistent and visually diverse results, outperforming RoboEngine yuan2025roboengine, which is a single-image-based method, and Cosmos-Transfer2.5 ali2025world, which struggles to generalize beyond appearance-level edge conditioning. Zoom in for the best view.
  • ...and 7 more figures