Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow

TL;DR

Dream2Flow introduces 3D object flow as a general interface that translates text-conditioned video predictions into executable robotic actions. By extracting 3D object trajectories from generated videos and framing manipulation as object trajectory tracking, the method decouples world state changes from embodiment and enables zero-shot grounding across rigid, articulated, deformable, and granular objects. The approach integrates video generation, depth estimation, and point tracking with planning via trajectory optimization or reinforcement learning, and demonstrates robustness and generalization in both simulation and real-world tasks. Key findings include the superiority of 3D object flow over rigid-transform baselines, the importance of video model choice and per-point dynamics, and the potential of flow-based rewards for learning sensimotor policies in diverse embodiments.

Abstract

Generative video modeling has emerged as a compelling tool to zero-shot reason about plausible physical interactions for open-world manipulation. Yet, it remains a challenge to translate such human-led motions into the low-level actions demanded by robotic systems. We observe that given an initial image and task instruction, these models excel at synthesizing sensible object motions. Thus, we introduce Dream2Flow, a framework that bridges video generation and robotic control through 3D object flow as an intermediate representation. Our method reconstructs 3D object motions from generated videos and formulates manipulation as object trajectory tracking. By separating the state changes from the actuators that realize those changes, Dream2Flow overcomes the embodiment gap and enables zero-shot guidance from pre-trained video models to manipulate objects of diverse categories-including rigid, articulated, deformable, and granular. Through trajectory optimization or reinforcement learning, Dream2Flow converts reconstructed 3D object flow into executable low-level commands without task-specific demonstrations. Simulation and real-world experiments highlight 3D object flow as a general and scalable interface for adapting video generation models to open-world robotic manipulation. Videos and visualizations are available at https://dream2flow.github.io/.

Figures (11)

• Figure 1: An overview of Dream2Flow. Given a task instruction and an initial RGB-D observation, an image-to-video model synthesizes video frames conditioned on the instruction. We additionally obtain object masks, video depth, and point tracking from vision foundation models, which are used to reconstruct 3D object flow. Finally, a robot policy generates executable actions that track the 3D object flow using trajectory optimization or reinforcement learning.
• Figure 2: Evaluation Tasks. (a) The initial states of each task used in the evaluation trials. (b) For one of the initial states, the corresponding final state after the robot performs the task. For Push-T only, the desired final state is the same for all initial states.
• Figure 3: Robustness evaluations. Relative performance across instance, background, and task variations, showing Dream2Flow remains robust under various different settings.
• Figure 4: Multiple tasks in the same scene. With different language goals, Dream2Flow adapts object-flow targets to produce distinct behaviors in the same environment.
• Figure 5: Rollouts from policies trained using 3D object flow as a reward. Different embodiments such as the (a) Panda, (b) Spot, or (c) GR1 use different strategies to open the door. The Spot is able move its base for better reachability while the GR1 uses the area between its fingers and palm to pull for better stability.