CogStereo: Neural Stereo Matching with Implicit Spatial Cognition Embedding

TL;DR

CogStereo tackles zero-shot generalization gaps in stereo matching by injecting implicit spatial cognition (SC) derived from monocular depth foundation models as priors, rather than dataset-specific priors. It introduces a Dual-Condition Refinement that combines pixelwise uncertainty with SC features to achieve globally coherent disparity in occluded/textureless regions. Key components include a Cost Volume Uncertainty Estimation prior to pre-training and a UG-SCA attention mechanism, along with LU-KSS and ADDG to maintain metric consistency and smoothness. Experiments across Scene Flow, KITTI, ETH3D, Middlebury, EuRoC, and real-world data show state-of-the-art results and strong zero-shot generalization, suggesting a cognition-driven shift in stereo matching.

Abstract

Deep stereo matching has advanced significantly on benchmark datasets through fine-tuning but falls short of the zero-shot generalization seen in foundation models in other vision tasks. We introduce CogStereo, a novel framework that addresses challenging regions, such as occlusions or weak textures, without relying on dataset-specific priors. CogStereo embeds implicit spatial cognition into the refinement process by using monocular depth features as priors, capturing holistic scene understanding beyond local correspondences. This approach ensures structurally coherent disparity estimation, even in areas where geometry alone is inadequate. CogStereo employs a dual-conditional refinement mechanism that combines pixel-wise uncertainty with cognition-guided features for consistent global correction of mismatches. Extensive experiments on Scene Flow, KITTI, Middlebury, ETH3D, EuRoc, and real-world demonstrate that CogStereo not only achieves state-of-the-art results but also excels in cross-domain generalization, shifting stereo vision towards a cognition-driven approach.

Figures (6)

• Figure 1: Illustration of the motivation behind CogStereo. While modern stereo networks achieve highly accurate disparity estimation in most regions, they remain vulnerable to mismatches in textureless, reflective, or occluded areas. CogStereo addresses these challenges by embedding implicit SC, enabling globally consistent and robust disparity estimation.
• Figure 2: Zero-shot prediction on EuRoC, demonstrating CogStereo's generalization to practical scenarios with challenging properties like weak texture, appearance ambiguities, reflectance, translucency and occlusion.
• Figure 3: Overview of Our CogStereo Framework. The CogStereo framework enhances stereo matching accuracy through two stages. Stage 1 pretrains the network to generate an uncertainty map from the cost volume. Stage 2 uses the UG-SCA module to refine disparities, leveraging SC to correct errors in high-uncertainty regions. The ConvGRU Block can be selected from an arbitrary neural stereo matching framework.
• Figure 4: Effectiveness of Uncertainty Masking on Disparity Estimation Accuracy. The top row shows the left image and disparity EPE map, highlighting high-error regions. Subsequent rows show reduced EPE with high uncertainty regions masked, where lower values mean higher accuracy.
• Figure 5: Qualitative comparison of zero-shot inference on in-the-wild images, including samples from the KITTI training dataset and our self-collected mid-range stereo matching dataset for autonomous driving. The red boxes highlight occlusion regions, the green boxes indicate reflection regions, and the yellow mark weak textures. CogStereo outperforms the baselines, demonstrating its effectiveness in handling such complexities.