ViPRA: Video Prediction for Robot Actions

TL;DR

ViPRA tackles learning robot policies from actionless videos by extracting motion-centric latent actions and grounding them in a multimodal video-language model. It pretrains to predict future visual observations and latent action sequences, and finetunes a flow-matching decoder to map latents to smooth, high-frequency robot actions using only hundreds of teleoperation demonstrations. The approach yields strong gains on SIMPLER and real-world manipulation, while generalizing across embodied agents due to diverse actionless video pretraining. This work outlines a scalable, data-efficient path for leveraging large-scale video data to build robust generalist policies for manipulation.

Abstract

Can we turn a video prediction model into a robot policy? Videos, including those of humans or teleoperated robots, capture rich physical interactions. However, most of them lack labeled actions, which limits their use in robot learning. We present Video Prediction for Robot Actions (ViPRA), a simple pretraining-finetuning framework that learns continuous robot control from these actionless videos. Instead of directly predicting actions, we train a video-language model to predict both future visual observations and motion-centric latent actions, which serve as intermediate representations of scene dynamics. We train these latent actions using perceptual losses and optical flow consistency to ensure they reflect physically grounded behavior. For downstream control, we introduce a chunked flow matching decoder that maps latent actions to robot-specific continuous action sequences, using only 100 to 200 teleoperated demonstrations. This approach avoids expensive action annotation, supports generalization across embodiments, and enables smooth, high-frequency continuous control upto 22 Hz via chunked action decoding. Unlike prior latent action works that treat pretraining as autoregressive policy learning, explicitly models both what changes and how. Our method outperforms strong baselines, with a 16% gain on the SIMPLER benchmark and a 13% improvement across real world manipulation tasks. We will release models and code at https://vipra-project.github.io

Figures (11)

• Figure 1: We present ViPRA, a framework to learn generalist robot policies from large-scale human and robot videos without action labels. It extracts motion-centric latent action sequences, pretrains a video-language model to jointly predict future visual observations and latent action chunks, and finetunes a flow matching decoder to map these latents to smooth, continuous action chunks for high-frequency, reactive control.
• Figure 2: ViPRA framework comprises of: (1) Latent Action Learning (left): A neural quantization bottleneck extracts discrete latent actions $z_t$ from image sequences $o_{0:L}$ in both human and robot videos, trained via reconstruction loss $\mathcal{L}_{\text{latent}}$ to capture motion-centric dynamics. (2) Multimodal Pretraining (center): A video-language model jointly predicts future observations $o_{t+H}$ and latent action sequences $z_{t:t+H-1}$ from past frames $(o_{t-1}, o_t)$ and task description $c$, using loss $\mathcal{L}_{\text{pretrain}}$. (3) Continuous Finetuning (right): A flow matching decoder maps latent actions to continuous robot actions $a_{t:t+H-1}$ using noisy action conditioning and loss $\mathcal{L}_{\text{FM}}$, enabling smooth, high-frequency control.
• Figure 3: Real World Evaluations (Left) We report full and partial success rates on three manipulation tasks. ViPRA-FM significantly outperforms baselines. (Right) We show our physical robot setup and task objects.
• Figure 4: Action smoothness: ViPRA-FM (blue) produces smooth, continuous trajectories, while LAPA (green) exhibits local discontinuities and random spikes, often around contact events, despite tracking the overall trend. During real world deployment, such discontinuities triggered the emergency brake mechanism of the robot due to abrupt motor torque jumps.
• Figure 5: Positional codebook usage differences across action categories. Each heatmap shows the difference in per-position token usage between two groups: (left) vertical vs. horizontal, (middle) left $\rightarrow$ right vs. right $\rightarrow$ left, and (right) up vs. down. ViPRA learns positionally sensitive codes, with certain entries (e.g., 0, 2, 5) showing systematic variation, indicating that both token index and positions encode action dynamics.