SemStereo: Semantic-Constrained Stereo Matching Network for Remote Sensing

TL;DR

SemStereo addresses the coupling gap between semantic segmentation and stereo matching in remote sensing by introducing a Semantic-Guided Cascade structure that implicitly shares deep features, plus explicit SSR and LRSC modules that refine disparities and enforce cross-view semantic consistency. The method uses a shared U-shaped encoder, a cascaded Fast-ACV-based disparity pipeline, and a semantic-aware refinement process with a joint loss that combines segmentation, disparity, and cross-view supervision. Empirical results on US3D and WHU demonstrate state-of-the-art performance for both tasks, with notable gains from semantic guidance, even in low-semantic-label scenarios, and strong generalization across cities. The work highlights the mutual benefits of semantic and disparity information for accurate semantic urban 3D reconstruction and points to future extensions to multi-view stereo and instance-level disparity modeling.

Abstract

Semantic segmentation and 3D reconstruction are two fundamental tasks in remote sensing, typically treated as separate or loosely coupled tasks. Despite attempts to integrate them into a unified network, the constraints between the two heterogeneous tasks are not explicitly modeled, since the pioneering studies either utilize a loosely coupled parallel structure or engage in only implicit interactions, failing to capture the inherent connections. In this work, we explore the connections between the two tasks and propose a new network that imposes semantic constraints on the stereo matching task, both implicitly and explicitly. Implicitly, we transform the traditional parallel structure to a new cascade structure termed Semantic-Guided Cascade structure, where the deep features enriched with semantic information are utilized for the computation of initial disparity maps, enhancing semantic guidance. Explicitly, we propose a Semantic Selective Refinement (SSR) module and a Left-Right Semantic Consistency (LRSC) module. The SSR refines the initial disparity map under the guidance of the semantic map. The LRSC ensures semantic consistency between two views via reducing the semantic divergence after transforming the semantic map from one view to the other using the disparity map. Experiments on the US3D and WHU datasets demonstrate that our method achieves state-of-the-art performance for both semantic segmentation and stereo matching.

Figures (6)

• Figure 1: A comparison between our and previous methods.
• Figure 2: The distribution of disparity per semantic category in satellite and ground-level perspectives, respectively.
• Figure 3: An overview of the SemStereo. It involves (a) a Semantic-Guided Cascade (SGC) structure for generating segmentation and initial disparity maps, (b) a Semantic Selective Refinement (SSR) branch refines the initial disparity under the guidance of semantic information, and (c) a Left-Right Semantic Consistency (LRSC) supervision. P: Prediction, GT: Ground Truth.
• Figure 4: Qualitative comparison of results with other state-of-the-art models on the US3D test set. Red box area: Our SemStereo achieves clearer boundaries in dense buildings; Black box area: Our model has clearer boundaries even for areas without disparity labels; Yellow box area: The prediction of our model preserves more details.
• Figure 5: Qualitative comparison of results for local details with other state-of-the-art models on the US3D test set. Only our SemStereo can effectively estimate the disparities of the signal tower and its surrounding small-scale object.