Future Optical Flow Prediction Improves Robot Control & Video Generation

TL;DR

FOFPred tackles the challenge of forecasting dense, language-conditioned future motion by introducing a unified Vision-Language Model and diffusion architecture to predict sequences of future optical flow from web-scale video-caption data. It encodes language and visuals via a Qwen2.5-VL VLM and a Flux.1 VAE, then uses a Diffusion Transformer with temporal RoPE and spatio-temporal attention to produce RGB-encoded optical flow targets that are learned with a diffusion loss and motion-disentangling supervision. The approach is validated on two downstream tasks: language-driven robot manipulation (CALVIN and RoboTwin) and motion-guided text-to-video generation (SSv2), with extensive ablations showing the value of web-scale pretraining, the unified backbone, and dense, camera-disentangled motion targets. The results demonstrate cross-domain benefits and highlight a scalable path toward motion-aware world models capable of guiding both control and generation across diverse, language-conditioned tasks.

Abstract

Future motion representations, such as optical flow, offer immense value for control and generative tasks. However, forecasting generalizable spatially dense motion representations remains a key challenge, and learning such forecasting from noisy, real-world data remains relatively unexplored. We introduce FOFPred, a novel language-conditioned optical flow forecasting model featuring a unified Vision-Language Model (VLM) and Diffusion architecture. This unique combination enables strong multimodal reasoning with pixel-level generative fidelity for future motion prediction. Our model is trained on web-scale human activity data-a highly scalable but unstructured source. To extract meaningful signals from this noisy video-caption data, we employ crucial data preprocessing techniques and our unified architecture with strong image pretraining. The resulting trained model is then extended to tackle two distinct downstream tasks in control and generation. Evaluations across robotic manipulation and video generation under language-driven settings establish the cross-domain versatility of FOFPred, confirming the value of a unified VLM-Diffusion architecture and scalable learning from diverse web data for future optical flow prediction.

Figures (4)

• Figure 1: Overview of Proposed FOFPred: (Left & Center) We present the unified VLM-Diffusion architecture used in FOFPred. Only the DiT module is trained while the VAE and VLM remain frozen. (Right) We illustrate two distinct pipelines constructed with FOFPred for two orthogonal tasks in control and generation. Each task specific head is first finetuned prior to inference on the downstream task.
• Figure 2: Relative Optical Flow Calculation: We illustrate the key stages of the algorithm for calculating optical flow targets for our training.
• Figure A.1: Sensitivity to seed: We visualize 4 FOFPred predictions for the same image and prompt, "Moving the bowl from left to right", but using 4 different starting noise vectors for the reverse diffusion process. Notice how the upper-right corner conflates the object motion with a camera motion instead; however this camera motion does correspond to the object motion described in the provided prompt. In the lower left image, in addition to the desired object motion, we again observe a slight amount of corresponding camera motion.
• Figure A.2: Visualization of success and failure cases for Text-to-Video (T2V) generation: We visualize some success and failure cases for our framework over the baseline, CogVideoX Yang2024CogVideoXTD. Examples are drawn from the SSv2 validation split. We note that our method consistently improves motion adherence over the baseline. However, in some cases our framework distorts the visual appearance of objects although they undergo correct movement (e.g., see "toy car" in Row 2). Checkout our https://fofpred.github.io for more visualizations.