Robust Incremental Structure-from-Motion with Hybrid Features

TL;DR

This work tackles robust SfM in challenging, low-texture scenes by introducing an end-to-end incremental SfM system that jointly leverages points, lines, vanishing points, and their structural relations. It extends the mapping, refinement, and registration stages to incorporate lines and VP constraints, along with principled uncertainty propagation for 3D lines via second-order sensitivity analysis, yielding richer maps and more reliable camera localization. The method demonstrates superior robustness and accuracy compared to the point-based COLMAP baseline on Hypersim, ETH3D, and PhotoTourism, and provides uncertainty-aware localization improvements on public benchmarks. By open-sourcing the implementation, the work aims to catalyze research and downstream applications that require robust, structure-aware 3D reconstructions.

Abstract

Structure-from-Motion (SfM) has become a ubiquitous tool for camera calibration and scene reconstruction with many downstream applications in computer vision and beyond. While the state-of-the-art SfM pipelines have reached a high level of maturity in well-textured and well-configured scenes over the last decades, they still fall short of robustly solving the SfM problem in challenging scenarios. In particular, weakly textured scenes and poorly constrained configurations oftentimes cause catastrophic failures or large errors for the primarily keypoint-based pipelines. In these scenarios, line segments are often abundant and can offer complementary geometric constraints. Their large spatial extent and typically structured configurations lead to stronger geometric constraints as compared to traditional keypoint-based methods. In this work, we introduce an incremental SfM system that, in addition to points, leverages lines and their structured geometric relations. Our technical contributions span the entire pipeline (mapping, triangulation, registration) and we integrate these into a comprehensive end-to-end SfM system that we share as an open-source software with the community. We also present the first analytical method to propagate uncertainties for 3D optimized lines via sensitivity analysis. Experiments show that our system is consistently more robust and accurate compared to the widely used point-based state of the art in SfM -- achieving richer maps and more precise camera registrations, especially under challenging conditions. In addition, our uncertainty-aware localization module alone is able to consistently improve over the state of the art under both point-alone and hybrid setups.

Figures (16)

• Figure 1: Incremental Structure-from-Motion with points, lines, and vanishing points.Red: groundtruths. Blue: predictions. We show example indoor scenes where classical point-based Structure-from-Motion fails. Leveraging additional constraints from line segments, our pipeline can faithfully reconstruct the scene and cameras.
• Figure 2: Our proposed SfM pipeline exploits hybrid features including points, lines, and vanishing points (VPs). We improve with technical contributions over all three main components: registration, triangulation, and refinement, leading to richer 3D maps (with uncertainty measurements) and more robust camera localization.
• Figure 3: Line triangulation is sensitive to view configurations. Left: unstable tracks from small baselines, few supports, or degenerate patterns. Right: an example stable track.
• Figure 4: Relations between the propagated uncertainty for each track and its accuracy on ETH3D sinha2012multi.Left: For each 3D feature, we plot its 3D error and uncertainty both in meters. Right: We report precision over each bin sorted w.r.t. to the 3D uncertainty. Points and lines with lower uncertainty tend to have higher precision.
• Figure 5: Some examples of our hybrid maps on Hypersim roberts:2021. Parallel lines from line-VP associations are colored the same.