Dense-SfM: Structure from Motion with Dense Consistent Matching

TL;DR

Dense-SfM presents a dense Structure-from-Motion pipeline that overcomes fragmentary multi-view tracks by integrating track extension via Gaussian Splatting with a multi-view refinement module based on a transformer and Gaussian Process. This approach yields denser and more accurate 3D reconstructions, particularly in texture-poor scenes, validated across ETH3D, Texture-Poor SfM, and IMC datasets. Key contributions include a three-stage workflow (dense-initial SfM, GS-based track extension, and iterative, learned refinement with confidence-aware regression) and extensive ablations demonstrating the superiority of the proposed matching and refinement components. The work advances practical dense SfM by enabling longer, more consistent feature tracks and more accurate camera poses, with tangible improvements in both accuracy and point density for challenging scenes.

Abstract

We present Dense-SfM, a novel Structure from Motion (SfM) framework designed for dense and accurate 3D reconstruction from multi-view images. Sparse keypoint matching, which traditional SfM methods often rely on, limits both accuracy and point density, especially in texture-less areas. Dense-SfM addresses this limitation by integrating dense matching with a Gaussian Splatting (GS) based track extension which gives more consistent, longer feature tracks. To further improve reconstruction accuracy, Dense-SfM is equipped with a multi-view kernelized matching module leveraging transformer and Gaussian Process architectures, for robust track refinement across multi-views. Evaluations on the ETH3D and Texture-Poor SfM datasets show that Dense-SfM offers significant improvements in accuracy and density over state-of-the-art methods. Project page: https://icetea-cv.github.io/densesfm/.

Figures (6)

• Figure 1: Comparison of SfM results between quantized-matching and our matching. To solve the fragmentary track problem, DFSfM detectorfreesfm uses quantized matching, which can compromise the accuracy and completeness of 3D points due to quantization. In contrast, our Dense-SfM pipeline leverages Gaussian Splatting to solve the problem, offering more accurate and dense 3D points. Thanks to dense matching, our pipeline can also be applied to texture-less objects, further improving the quality of 3D points and pose estimation.
• Figure 2: Pipeline Overview. From a set of images, we construct an initial SfM model using dense two-view matching, filtering unreliable matches through mutual verification. To extend track length, we project 3D points onto additional images, using a visibility filter based on Gaussian Splatting. We then refine these extended tracks with our track refinement module and perform geometric bundle adjustment to improve the accuracy of SfM model.
• Figure 3: Bidirectional verification on two-view dense matching. The match result $p_b$ from $\mathbf{\mathcal{M}}_{A\rightarrow{} B}$ is re-used as input to $\mathbf{\mathcal{M}}_{B\rightarrow{} A}$ to estimate $p_{a'}$. We then compute the distance between $p_a$ and $p_{a'}$, where a smaller distance indicates higher reliability of the match between $p_a$ and $p_b$.
• Figure 4: Multi-view kernelized matching architecture. The feature maps from the reference and query views are processed in two ways: first, by a multi-view transformer, applying self and cross-attention and secondly, by a Gaussian Process that computes the posterior mean conditioned on the feature of each query view. All results are concatenated and fed into a CNN decoder, yielding probability distribution of coordinates for refinement $\mathbf{P}_{Q_i}$ and corresponding confidence score $\mathbf{S}_{Q_i}$ for each query view.
• Figure 5: Regression of refined coordinate.Top: Selection of refined coordinate on the reference view: From the set of $S_{Q_i}$ values, representing the correspondence scores from the reference view to each query view $Q_i$, we sum over $S_{Q_i}$'s and select the pixel with the highest score as the refined coordinate for the reference view. Bottom: Regression of refined coordinate on each query view: We apply a channel-wise softmax to obtain a probability distribution of coordinates for each pixel, then calculate a weighted average to determine the correspondence from the reference view to query view $Q_i$.