Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Boosting Zero-shot Stereo Matching using Large-scale Mixed Images Sources in the Real World

Yuran Wang, Yingping Liang, Ying Fu

TL;DR

The paper tackles the limited labeled data and domain gaps that hinder stereo matching, especially in real-world settings. It introduces BooSTer, a framework that combines monocular-depth-guided stereo data generation from single-view images using diffusion models, pseudo-label supervision with a dynamic scale- and shift-invariant loss, and a hybrid feature encoder that fuses vision foundation model representations with conventional CNN features. Key contributions include a monocular-to-stereo data generation pipeline (DiffMFS), the DSSI loss for aligning pseudo-depth with stereo predictions, and a VFM-based encoder to improve cross-domain generalization. Experiments show BooSTer delivers significant improvements in zero-shot generalization and robustness under domain shifts on standard benchmarks, highlighting its practical impact for real-world stereo applications.

Abstract

Stereo matching methods rely on dense pixel-wise ground truth labels, which are laborious to obtain, especially for real-world datasets. The scarcity of labeled data and domain gaps between synthetic and real-world images also pose notable challenges. In this paper, we propose a novel framework, \textbf{BooSTer}, that leverages both vision foundation models and large-scale mixed image sources, including synthetic, real, and single-view images. First, to fully unleash the potential of large-scale single-view images, we design a data generation strategy combining monocular depth estimation and diffusion models to generate dense stereo matching data from single-view images. Second, to tackle sparse labels in real-world datasets, we transfer knowledge from monocular depth estimation models, using pseudo-mono depth labels and a dynamic scale- and shift-invariant loss for additional supervision. Furthermore, we incorporate vision foundation model as an encoder to extract robust and transferable features, boosting accuracy and generalization. Extensive experiments on benchmark datasets demonstrate the effectiveness of our approach, achieving significant improvements in accuracy over existing methods, particularly in scenarios with limited labeled data and domain shifts.

Boosting Zero-shot Stereo Matching using Large-scale Mixed Images Sources in the Real World

TL;DR

The paper tackles the limited labeled data and domain gaps that hinder stereo matching, especially in real-world settings. It introduces BooSTer, a framework that combines monocular-depth-guided stereo data generation from single-view images using diffusion models, pseudo-label supervision with a dynamic scale- and shift-invariant loss, and a hybrid feature encoder that fuses vision foundation model representations with conventional CNN features. Key contributions include a monocular-to-stereo data generation pipeline (DiffMFS), the DSSI loss for aligning pseudo-depth with stereo predictions, and a VFM-based encoder to improve cross-domain generalization. Experiments show BooSTer delivers significant improvements in zero-shot generalization and robustness under domain shifts on standard benchmarks, highlighting its practical impact for real-world stereo applications.

Abstract

Stereo matching methods rely on dense pixel-wise ground truth labels, which are laborious to obtain, especially for real-world datasets. The scarcity of labeled data and domain gaps between synthetic and real-world images also pose notable challenges. In this paper, we propose a novel framework, \textbf{BooSTer}, that leverages both vision foundation models and large-scale mixed image sources, including synthetic, real, and single-view images. First, to fully unleash the potential of large-scale single-view images, we design a data generation strategy combining monocular depth estimation and diffusion models to generate dense stereo matching data from single-view images. Second, to tackle sparse labels in real-world datasets, we transfer knowledge from monocular depth estimation models, using pseudo-mono depth labels and a dynamic scale- and shift-invariant loss for additional supervision. Furthermore, we incorporate vision foundation model as an encoder to extract robust and transferable features, boosting accuracy and generalization. Extensive experiments on benchmark datasets demonstrate the effectiveness of our approach, achieving significant improvements in accuracy over existing methods, particularly in scenarios with limited labeled data and domain shifts.
Paper Structure (17 sections, 9 equations, 5 figures, 3 tables)

This paper contains 17 sections, 9 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Stereo Matching Method
  4. Stereo Datasets
  5. Vision Foundation Models
  6. Method
  7. Motivation and Formulation
  8. Learning from Single-view Images
  9. Learning from Sparse Images
  10. VFM Based Hybrid Feature Encoder
  11. Experiment
  12. Datasets and Evaluation Metrics
  13. Implementation Details
  14. Comparison of Zero-Shot Generalization
  15. Comparison of Data Generation Methods
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Comparison between Monocular Depth and Stereo Matching. Monocular Depth approach provides fine-grained details but suffers from scale ambiguity, whereas Stereo Matching delivers real-world metrics but with coarser results. We carefully design data generation and dynamic scale- and shift- invariant loss to perform knowledge transfer from monocular to stereo.
  • Figure 2: The overall architecture of our proposed method consists of two main parts. 1. Large-scale multi-source mixed training dataset. We mix stereo data from various scenarios as a large-scale pre-training dataset. 2. VFM Stereo Matching Model. We propose a hybrid encoder structure embedded in VFM to boost the generalization and performance of the algorithm by transferring existing knowledge.
  • Figure 3: Comparison between naive inpainting module and our inpainting module. (a) warped right-view image with occulusion holes, (b) image inpainted from (a) using SD, and (c) right image warping with our inpainting module.
  • Figure 4: Comparison between disparity from KITTI and from monocular model. (a) RGB Image, (b) ground-truth from LiDAR, and (c) disparity from monocular estimation.
  • Figure 5: Qualitative results of IGEV and StereoBase trained with Sceneflow and our model trained on our mixed dataset. By default, IGEV and StereoBase are trained using SceneFlow.