VLM-SFD: VLM-Assisted Siamese Flow Diffusion Framework for Dual-Arm Cooperative Manipulation

TL;DR

Dual-arm manipulation remains challenging to generalize from limited demonstrations. The authors propose VLM-SFD, a diffusion-based framework that learns two synchronized object-centric motion flows via a Siamese Flow Diffusion Network and assigns them to arms with a VLM-guided spatial-temporal allocator, enabling collision-free coordination. The SFDNet employs a Siamese VAE encoder, Siamese UNet, and Siamese VAE decoder conditioned on task instructions, producing two compatible flow streams that map to 3D trajectories for execution. Real-world experiments on four tasks with only ten demonstrations per task demonstrate superior performance over baseline methods and direct deployability without fine-tuning, highlighting data-efficient learning for practical dual-arm manipulation.

Abstract

Dual-arm cooperative manipulation holds great promise for tackling complex real-world tasks that demand seamless coordination and adaptive dynamics. Despite substantial progress in learning-based motion planning, most approaches struggle to generalize across diverse manipulation tasks and adapt to dynamic, unstructured environments, particularly in scenarios involving interactions between two objects such as assembly, tool use, and bimanual grasping. To address these challenges, we introduce a novel VLM-Assisted Siamese Flow Diffusion (VLM-SFD) framework for efficient imitation learning in dual-arm cooperative manipulation. The proposed VLM-SFD framework exhibits outstanding adaptability, significantly enhancing the ability to rapidly adapt and generalize to diverse real-world tasks from only a minimal number of human demonstrations. Specifically, we propose a Siamese Flow Diffusion Network (SFDNet) employs a dual-encoder-decoder Siamese architecture to embed two target objects into a shared latent space, while a diffusion-based conditioning process - conditioned by task instructions - generates two-stream object-centric motion flows that guide dual-arm coordination. We further design a dynamic task assignment strategy that seamlessly maps the predicted 2D motion flows into 3D space and incorporates a pre-trained vision-language model (VLM) to adaptively assign the optimal motion to each robotic arm over time. Experiments validate the effectiveness of the proposed method, demonstrating its ability to generalize to diverse manipulation tasks while maintaining high efficiency and adaptability. The code and demo videos are publicly available on our project website https://sites.google.com/view/vlm-sfd/.

Figures (4)

• Figure 1: Overview of VLM-SFD Framework. In the training stage, demonstration video and task instruction are first fed into Grounding-DINO dino and TAPIR tapir for localizing the objects to be manipulated and extracting corresponding flow, which are then used for training the Siamese Flow Diffusion Network (SFDNet) to predict dense 2D motion flows. In the testing stage, given an initial frame and an instruction, the model predicts 2D flows, which are then post-processed into 3D trajectories, which are further fed into the Spatial-Temporal Task Allocation module to generate coordinated dual-arm motion plans for robot execution.
• Figure 2: Task Examples and Execution Results. We evaluate our framework on four challenging tasks: (1) Put X into Pot (first row), (2) Packing (second row), (3) Pouring (third row), and (4) Pulling Drawer & Placing (fourth row). For each task, the figure shows the initial scene, flow generation visualizations, and sequential snapshots of robot execution.
• Figure 3: Snapshot of the real-world experimental setup: two Franka Research 3 robotic manipulators positioned along with an external Intel RealSense D435i RGB-D camera.
• Figure 4: Comparison of task success rates (%) for the proposed VLM-SFD framework vs. baseline methods ACT (blue) and DP (purple) across four dual-arm manipulation tasks. VLM-SFD (orange bars) achieves the highest success rate in every task by a wide margin. All success rates are averaged over 20 repeated trials per task.