From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation

TL;DR

This work tackles generalization in robotic manipulation by introducing FSD (From Seeing to Doing), a framework that generates intermediate visual aids through Spatial Relationship-Focused CoT (SrCoT) to bridge Vision-Language understanding with embodied action. It combines a five-level weak-to-strong data pipeline and a self-consistency mechanism to ground spatial coordinates to visual signals, enabling zero-shot manipulation and strong performance across multiple benchmarks including VABench. Empirically, FSD achieves a 40.6% success rate in SimplerEnv and 72% across eight real-world tasks, outperforming baselines by about 30% and surpassing prior VLA methods in visual-aid generation. The approach reduces reliance on homogeneous embodied data by leveraging coordinate-based spatial representations and broad visual data, with future work aiming to extend to 3D visual aids and long-horizon instructions in $SE(3)$ space.

Abstract

Achieving generalization in robotic manipulation remains a critical challenge, particularly for unseen scenarios and novel tasks. Current Vision-Language-Action (VLA) models, while building on top of general Vision-Language Models (VLMs), still fall short of achieving robust zero-shot performance due to the scarcity and heterogeneity prevalent in embodied datasets. To address these limitations, we propose FSD (From Seeing to Doing), a novel vision-language model that generates intermediate representations through spatial relationship reasoning, providing fine-grained guidance for robotic manipulation. Our approach combines a hierarchical data pipeline for training with a self-consistency mechanism that aligns spatial coordinates with visual signals. Through extensive experiments, we comprehensively validated FSD's capabilities in both "seeing" and "doing," achieving outstanding performance across 8 benchmarks for general spatial reasoning and embodied reference abilities, as well as on our proposed more challenging benchmark VABench. We also verified zero-shot capabilities in robot manipulation, demonstrating significant performance improvements over baseline methods in both SimplerEnv and real robot settings. Experimental results show that FSD achieves 40.6% success rate in SimplerEnv and 72% success rate across 8 real-world tasks, outperforming the strongest baseline by 30%.

Figures (15)

• Figure 1: Overview of FSD. FSD unlocks visual aids reasoning and generation through Spatial Relationship-Focused CoT, demonstrating exceptional generalization capabilities that enable zero-shot robot manipulation and achieving remarkable performance across multiple benchmarks.
• Figure 2: Diagrams of Visual Aid Types
• Figure 3: Inspired by the process of human reasoning, FSD uses a spatial relationship graph as an anchor to derive a visual chain-of-thought reasoning process for visual trace generation.
• Figure 4: FSD screens data from large-scale embodied datasets, generates ground truth spatial relationship graph. We finally collected 300K data for 10+ embodiments with 5-level capabilities.
• Figure 5: FSD directly generates visual aids based on task instructions for novel tasks and scenarios. 1st row: affordance bounding boxes; 2nd row: affordance points; 3rd and 4th rows: visual traces.