Flow as the Cross-Domain Manipulation Interface

TL;DR

Im2Flow2Act tackles scalable real-world robot manipulation by bridging cross-domain data via object flow as a unifying interface. It combines a language-conditioned flow generation network trained on human demonstrations with a flow-conditioned policy learned in simulation to map generated flows to robot actions, significantly reducing the sim-to-real gap and removing the need for real-robot training data. The approach achieves an average 81% real-world success across four tasks, including rigid, articulated, and deformable objects, and outperforms strong baselines across both demonstration- and language-conditioned settings. This work demonstrates a practical, data-efficient pathway for acquiring diverse robotic manipulation skills from heterogeneous data sources.

Abstract

We present Im2Flow2Act, a scalable learning framework that enables robots to acquire real-world manipulation skills without the need of real-world robot training data. The key idea behind Im2Flow2Act is to use object flow as the manipulation interface, bridging domain gaps between different embodiments (i.e., human and robot) and training environments (i.e., real-world and simulated). Im2Flow2Act comprises two components: a flow generation network and a flow-conditioned policy. The flow generation network, trained on human demonstration videos, generates object flow from the initial scene image, conditioned on the task description. The flow-conditioned policy, trained on simulated robot play data, maps the generated object flow to robot actions to realize the desired object movements. By using flow as input, this policy can be directly deployed in the real world with a minimal sim-to-real gap. By leveraging real-world human videos and simulated robot play data, we bypass the challenges of teleoperating physical robots in the real world, resulting in a scalable system for diverse tasks. We demonstrate Im2Flow2Act's capabilities in a variety of real-world tasks, including the manipulation of rigid, articulated, and deformable objects.

Figures (11)

• Figure 1: Flow as the Cross-domain Manipulation Interface. In Im2Flow2Act, we utilize object flow to bridge the domain gap between both embodiments (human v.s. robot) and training environments (real v.s. simulation). Our final system is able to leverage both a) action-less human video for task-conditioned flow generation and b) task-less simulated robot data for flow conditioned action generation, results in c)a language-conditioned multi-task system for real-world manipulation tasks but without the need of real-robot training data.
• Figure 2: Im2Flow2Act Overview. Given a task description and the initial frame, the flow generator (a) generates a complete object flow for the task (i.e., task flow). The closed-loop manipulation policy (c), takes in the task flow and the keypoint location at the current frame to infer the robot actions. Within the manipulation policy, a temporal alignment module (c-1) is used compare task flow and keypoint location at the current frame to infer remaining task flow (flow in color) for the diffusion head (c-3) to generate actions.
• Figure 3: Flow Generation Network. The flow generation network outputs object flow for the complete task conditioned on the initial frame and task description. The object of interest is first detected using Grounding DINO (a), then we sample grid points inside the bounding box as the initial keypoints. AnimateDiff (b) takes in keypoints and task description to generates the future flow. We postprocess the generated flow through a motion filter to obtain an object-centric task flow.
• Figure 4: Task and results: We conducted evaluations on four tasks involving rigid, articulated, and deformable objects. The initial scene, flow generation visualizations, and online point tracking during inference were captured using the RealSense camera. Visualizations of the robot’s execution were recorded from a different angle to provide a clearer view of the robot execution.
• Figure 5: Typical failure cases of baselines. Even given the ground truth flows produced by human, the heuristic policy may produce inaccurate actions, for example, pushing the drawer back first. As for the No Alignment model, the cup may be randomly rotated along the trajectory, which is the same behavior as the random exploration data.