VO-DP: Semantic-Geometric Adaptive Diffusion Policy for Vision-Only Robotic Manipulation

TL;DR

This work tackles the challenge of vision-only visuomotor manipulation by proposing VO-DP, a diffusion-policy framework that fuses semantic and geometric cues from a single RGB view using pretrained visual foundation models. The method combines a geometry-prior visual encoder (VGGT with DINOv2 and Alternating Attention), a cross-attention based fusion module, a CNN-based spatial compressor, and a DDPM-based action generator, enabling robust, data-efficient learning without depth sensors. In simulation, VO-DP nearly matches point-cloud based DP3 and outperforms vision-only DP, while in real-world tests it achieves the highest average success rate among baselines and demonstrates strong robustness to color, size, lighting, and background variations. The authors also release an open-source training library to facilitate scalable, multi-GPU training, highlighting the practical potential of vision-only diffusion policies for cost-effective, real-world robotic manipulation.

Abstract

In the context of imitation learning, visuomotor-based diffusion policy learning is one of the main directions in robotic manipulation. Most of these approaches rely on point clouds as observation inputs and construct scene representations through point clouds feature learning, which enables them to achieve remarkable accuracy. However, the existing literature lacks an in-depth exploration of vision-only solutions that have significant potential. In this paper, we propose a Vision-Only and single-view Diffusion Policy learning method (VO-DP) that leverages pretrained visual foundation models to achieve effective fusion of semantic and geometric features. We utilize intermediate features from VGGT incorporating semantic features from DINOv2 and geometric features from Alternating Attention blocks. Features are fused via cross-attention and spatially compressed with a CNN to form the input to the policy head. Extensive experiments demonstrate that VO-DP not only outperforms the vision-only baseline DP significantly but also exhibits distinct performance trends against the point cloud-based method DP3: in simulation tasks, VO-DP achieves an average success rate of 64.6% on par with DP3 64.0% and far higher than DP 34.8%, while in real-world tasks, it reaches 87.9%, outperforming both DP3 67.5% and DP 11.2% by a notable margin. Further robustness evaluations confirm that VO-DP remains highly stable under varying conditions including color, size, background, and lighting. Lastly, we open-source a training library for robotic manipulation. Built on Accelerate, this library supports multi-machine and multi-GPU parallel training, as well as mixed precision training. It is compatible with visuomotor policies such as DP, DP3 and VO-DP, and also supports the RoboTwin simulator.

Figures (13)

• Figure 1: VO-DP is a vision-only method for visuomotor robotic manipulation: it takes single-view RGB images as input, uses large vision models to extract semantic and geometric features from observations, and provides high-quality conditional inputs for the policy head. Experiments show it matches point cloud-based DP3’s accuracy in simulation, outperforms it significantly in real-world tasks, and notably boosts vision-only method accuracy.
• Figure 2: Overall architecture of VO-DP. VO-DP has four core modules: 1) VGGT Encoder extracts semantic features from patchified images via DINOv2 and generates geometric features through its AA network; 2) Semantic-Geometric Fuser fuses per-frame geometric and semantic features using residual cross-attention and an FFN; 3) Spatial Compression module reshapes fused features, downsamples them with a lightweight ResNet, and concatenates the compressed spatial features with proprioceptive observations to form compact scenario representations; 4) Vision-Only Conditioned Action Generation module employs a DDPM-based policy head to generate actions using the scenario representations.
• Figure 3: Simulation benchmark -- 14 bimanual manipulation tasks. Left: Top-view RGB image of the task. Right: Reconstructed point clouds via VGGT.
• Figure 4: Data efficiency and scaling capability comparison of VO-DP with baseline methods (DP and DP3) across four tasks. The table presents success rate changes as training demonstrations scale from 20 to 100, highlighting VO-DP’s more substantial performance improvements.
• Figure 5: Visualize of 4 real-world tasks: Pick&Place Small Cube (PPSC), Pick&Place Big Cube (PPBC), Cover Cuboid (CC), Stack Cubes (SC).