SED-MVS: Segmentation-Driven and Edge-Aligned Deformation Multi-View Stereo with Depth Restoration and Occlusion Constraint

TL;DR

SED-MVS tackles the problem of reconstructing dense 3D structure from multi-view images in textureless and edge-rich scenes by introducing segmentation-driven, edge-aligned patch deformation. It combines SAM2-based depth-edge guidance, a multi-trajectory diffusion strategy, and sparse-monocular synergistic restoration (LoFTR + DepthAnything V2) to initialize and supervise PatchMatch-like optimization, with occlusion-aware edge constraints guided by segmentation-derived occlusion maps. The approach yields state-of-the-art performance on ETH3D, Tanks & Temples, BlendedMVS, and Strecha, while demonstrating strong generalization and robustness to textureless regions and occlusions. This work advances MVS by integrating segmentation priors, edge alignment, and cross-view depth reconstruction, enabling more reliable 3D reconstruction in challenging real-world scenes.

Abstract

Recently, patch-deformation methods have exhibited significant effectiveness in multi-view stereo owing to the deformable and expandable patches in reconstructing textureless areas. However, such methods primarily emphasize broadening the receptive field in textureless areas, while neglecting deformation instability caused by easily overlooked edge-skipping, potentially leading to matching distortions. To address this, we propose SED-MVS, which adopts panoptic segmentation and multi-trajectory diffusion strategy for segmentation-driven and edge-aligned patch deformation. Specifically, to prevent unanticipated edge-skipping, we first employ SAM2 for panoptic segmentation as depth-edge guidance to guide patch deformation, followed by multi-trajectory diffusion strategy to ensure patches are comprehensively aligned with depth edges. Moreover, to avoid potential inaccuracy of random initialization, we combine both sparse points from LoFTR and monocular depth map from DepthAnything V2 to restore reliable and realistic depth map for initialization and supervised guidance. Finally, we integrate segmentation image with monocular depth map to exploit inter-instance occlusion relationship, then further regard them as occlusion map to implement two distinct edge constraint, thereby facilitating occlusion-aware patch deformation. Extensive results on ETH3D, Tanks & Temples, BlendedMVS and Strecha datasets validate the state-of-the-art performance and robust generalization capability of our proposed method.

Figures (7)

• Figure 1: Comparative analysis between ACMMP, APD-MVS, SD-MVS and our method. Green and red dots denote the central pixel and boundary pixel, respectively, while the red background indicates the patch. ACMMP in (a) struggles to reconstruct textureless areas as its fixed-size patch fails to capture sufficient feature points. The ignorance of depth edge causes deformed patch of APD-MVS in (b) to be selected in depth-discontinuous areas, leading to potential matching distortions. SD-MVS in (c) adjusts the patch scale based on the distance from pixel to the depth edge boundary, while it remains constrained by the fixed patch size and lacks edge alignment capabilities. In contrast, our method in (d) adopts panoptic segmentation and multi-trajectory diffusion to enable segmentation-driven and edge-aligned patch deformation.
• Figure 2: An illustrated pipeline of our method. Given the input images, we first adopt LoftR, DepthAnything V2 and SAM2 to obtain their sparse points, monocular depth maps and segmentation images, respectively. Then we regard segmentation images as the depth-edge guidance to accurately constrain the patch deformation within depth-continuous areas, followed by multi-trajectory diffusion to enable edge-aligned patch deformation. The deformed patches are then sequentially processed through texture-aware mapping, load balancing, and spherical gradient refinement to achieve matching cost, propagation, and refinement, respectively. Moreover, we adopt segmentation-driven triangulation and geometry-aware refinement to combine sparse points and monocular depth map, thereby generating restored depth map for initialization and supervised guidance during the PM process. Finally, we integrate segmentation image with monocular depth map to generate occlusion map for dual-category edge constraint, thereby enabling occlusion-aware patch deformation.
• Figure 3: Segmentation-driven and edge-aligned patch deformation. In (b), different colors represent different instance masks. From (c) to (e), higher color temperatures indicate greater depth values. From (f) to (j), instance edges are highlighted in blue, with green dots and red background respectively denoting the center pixels and the deformed patch. In (g), SD-MVS calculates distances in the vertical and horizontal directions from the center pixel to its boundaries, then proportionally deforms its patch accordingly. In (h), red dots and black lines respectively indicate boundary pixels and paths. In (i), cyan and blue dots respectively denote the initially selected pixels and the final mapping pixels for matching cost. For clarity, the texture-aware mapping strategy is only visualized in the white box. In (j), diagonal paths are grouped and marked with the same color. From each grouped path, one purple point is selected for propagation.
• Figure 4: Sparse-monocular synergistic restoration. In (b) and (c), feature point pairs are linked with lines of different colors. In (d), different colors represent distinct instance masks. In (e), lower color temperatures indicate greater depth values. In (f), instance edges and generated triangles are respectively highlighted in blue and red. Compared with the final depth map in (h), the restored depth map in (g) effectively recovers textureless areas without severe detail distortion.
• Figure 5: Occlusion-aware patch deformation. In (b), different colors represent different instance masks. In (c) and (f), higher color temperatures indicate greater depth values. Differently, in (e), lower color temperatures indicate greater depth values. In (d), blue and red edges respectively denote depth-continuous and depth-discontinuous boundaries. Compared with (c), depth map with occlusion constraint in (f) can effectively distinguish depth edges in textureless areas.