{S\textsuperscript{2}M\textsuperscript{2}}: Scalable Stereo Matching Model for Reliable Depth Estimation

TL;DR

S²M² tackles the generalization gap in stereo depth estimation by reimagining stereo as scalable global matching. It combines a multi-resolution transformer-based feature extractor with a global matching module solved via entropy-regularized optimal transport, followed by RAFT-like refinement and edge-aware upsampling. A novel Probabilistic Mode Concentration loss concentrates matching probability on feasible disparities, enabling reliable disparity, occlusion, and confidence estimation. The approach achieves state-of-the-art results on ETH3D and Middlebury v3, maintains strong performance across high-resolution and large-disparity scenarios, and offers a favorable accuracy-efficiency Pareto frontier compared to prior global or iterative methods. This framework advances generalizable, high-fidelity depth estimation suitable for safety-critical applications by balancing scalability, accuracy, and reliability.

Abstract

The pursuit of a generalizable stereo matching model, capable of performing well across varying resolutions and disparity ranges without dataset-specific fine-tuning, has revealed a fundamental trade-off. Iterative local search methods achieve high scores on constrained benchmarks, but their core mechanism inherently limits the global consistency required for true generalization. However, global matching architectures, while theoretically more robust, have historically been rendered infeasible by prohibitive computational and memory costs. We resolve this dilemma with {S\textsuperscript{2}M\textsuperscript{2}}: a global matching architecture that achieves state-of-the-art accuracy and high efficiency without relying on cost volume filtering or deep refinement stacks. Our design integrates a multi-resolution transformer for robust long-range correspondence, trained with a novel loss function that concentrates probability on feasible matches. This approach enables a more robust joint estimation of disparity, occlusion, and confidence. {S\textsuperscript{2}M\textsuperscript{2}} establishes a new state of the art on Middlebury v3 and ETH3D benchmarks, significantly outperforming prior methods in most metrics while reconstructing high-quality details with competitive efficiency.

Figures (4)

• Figure 1: Comprehensive evaluation on two representative benchmarks, ETH3D schops2017multi and Middlebury v3 scharstein2014high. Accuracy is measured using the Bad-2.0 metric for Middlebury and Bad-0.5 for ETH3D. Each method is plotted with its corresponding model size (in millions of parameters).
• Figure 2: Our proposed architecture, S2M², comprises four main stages: (1) feature extraction, where left and right input images are processed by FPN and MRT to yield multi-resolution feature maps; (2) global matching, which computes all-pairs correlation at 1/4 resolution and estimates disparity, occlusion, and confidence using optimal transport; (3) refinement, iteratively updating these estimates; and (4) upsampling, enhancing depth boundaries to restore full resolution. The feature extraction stage employs a Multi-Resolution Transformer (MRT) for hierarchical feature extraction, incorporating an Adaptive Gated Fusion Layer (AGFL) for efficient multi-scale feature alignment.
• Figure 3: Illustration of Probabilistic Mode Concentration (PMC) Loss. (a) Disparity map with a dashed box highlighting a 4×4 region, which corresponds to a single pixel in the 1/4 downsampled feature space. (b) Matching probability distribution at the selected location, where PMC Loss ensures that the probability mass remains concentrated within valid disparity candidates.
• Figure 4: Scalability Analysis