StereoVLA: Enhancing Vision-Language-Action Models with Stereo Vision

TL;DR

StereoVLA addresses the limitations of monocular inputs in Vision-Language-Action models by exploiting stereo geometry to enhance spatial perception for manipulation. It introduces a Geometric-Semantic Feature Extraction module that fuses FoundationStereo-based geometric cues with semantic tokens from SigLIP and DINOv2, plus an auxiliary Interaction-Region Depth Estimation co-training task. The approach yields large-magnitude improvements over baselines under stereo configurations and demonstrates robustness to camera pose variations, with ablations validating design choices. This work provides practical guidance for deploying stereo sensing in embodied multimodal learning and advances precision-critical robotic manipulation.

Abstract

Stereo cameras closely mimic human binocular vision, providing rich spatial cues critical for precise robotic manipulation. Despite their advantage, the adoption of stereo vision in vision-language-action models (VLAs) remains underexplored. In this work, we present StereoVLA, a VLA model that leverages rich geometric cues from stereo vision. We propose a novel Geometric-Semantic Feature Extraction module that utilizes vision foundation models to extract and fuse two key features: 1) geometric features from subtle stereo-view differences for spatial perception; 2) semantic-rich features from the monocular view for instruction following. Additionally, we propose an auxiliary Interaction-Region Depth Estimation task to further enhance spatial perception and accelerate model convergence. Extensive experiments show that our approach outperforms baselines by a large margin in diverse tasks under the stereo setting and demonstrates strong robustness to camera pose variations.

Figures (7)

• Figure 1: Common camera setups for robotic manipulation. (a) Single-view RGB suffers from depth ambiguity; (b) depth sensors yield noisy estimates for transparent objects; (c) wrist-view RGB captures a limited view and the additional hardware increases collision risk; (d) multi-camera settings incur additional deployment overhead. (e) Stereo RGB pairs provide robust spatial cues for comprehensive scene understanding, mitigate view limit and collision risk, while enabling a simpler setup with a single stereo camera.
• Figure 2: (a) In StereoVLA, a stereo image pair is encoded by the Geometric-Semantic Feature Extraction module to generate visual tokens with geometric precision and semantic richness. Together with language tokens, they are processed by a large language model backbone (InternLM-1.8B). An action expert predicts delta end-effector poses, while an auxiliary depth estimation task further enhances geometry learning during training. (b) The Geometric-Semantic Feature Extraction module extracts geometric features with FoundationStereo (bypassing disparity prediction components for efficiency) and semantic-rich features with SigLIP and DINOv2, then fuses them into a unified visual representation with an MLP projector.
• Figure 3: Real-world evaluation tasks and results. We evaluate StereoVLA on a comprehensive suite of tasks requiring fine-grained perception and manipulation, including grasping bar-shaped objects at various orientations, grasping objects of medium and small sizes, and general-purpose pick-and-place scenarios. Objects enclosed by colored boxes denote the target objects used for evaluation, while other items in the scene serve as distractors. StereoVLA consistently achieves the highest success rate across all task types, highlighting the effectiveness of our method.
• Figure 4: Layout of the robot arm, workspace, and cameras. To study method robustness to viewpoint changes, we define three randomization ranges for the front and side cameras.
• Figure 5: Comparison of feature fusion methods. Sequence concatenation results in a slightly lower success rate and requires a longer training time.