InstantSfM: Fully Sparse and Parallel Structure-from-Motion

TL;DR

InstantSfM tackles the scalability bottleneck of global Structure-from-Motion by deploying a fully sparse, GPU-accelerated pipeline implemented in PyTorch that unifies global positioning and bundle adjustment. It extends sparse-aware optimization to both GP and BA, enabling parallel updates on a single GPU with efficient memory reuse and robust outlier handling. Incorporating a depth prior yields metric-scale reconstructions and enhances robustness across large datasets, achieving up to ~40x speedups over COLMAP while maintaining, or improving, accuracy. Extensive experiments across MipNeRF360, DTU, Tanks and Temples, and ScanNet demonstrate strong performance and robustness, highlighting InstantSfM's practicality for robotics and simulation tasks requiring fast, large-scale 3D reconstruction.

Abstract

Structure-from-Motion (SfM), a method that recovers camera poses and scene geometry from uncalibrated images, is a central component in robotic reconstruction and simulation. Despite the state-of-the-art performance of traditional SfM methods such as COLMAP and its follow-up work, GLOMAP, naive CPU-specialized implementations of bundle adjustment (BA) or global positioning (GP) introduce significant computational overhead when handling large-scale scenarios, leading to a trade-off between accuracy and speed in SfM. Moreover, the blessing of efficient C++-based implementations in COLMAP and GLOMAP comes with the curse of limited flexibility, as they lack support for various external optimization options. On the other hand, while deep learning based SfM pipelines like VGGSfM and VGGT enable feed-forward 3D reconstruction, they are unable to scale to thousands of input views at once as GPU memory consumption increases sharply as the number of input views grows. In this paper, we unleash the full potential of GPU parallel computation to accelerate each critical stage of the standard SfM pipeline. Building upon recent advances in sparse-aware bundle adjustment optimization, our design extends these techniques to accelerate both BA and GP within a unified global SfM framework. Through extensive experiments on datasets of varying scales (e.g. 5000 images where VGGSfM and VGGT run out of memory), our method demonstrates up to about 40 times speedup over COLMAP while achieving consistently comparable or even improved reconstruction accuracy. Our project page can be found at https://cre185.github.io/InstantSfM/.

Figures (5)

• Figure 1: Qualitative results on various datasets. Camera frustums as red are visualized for illustrating the estimated camera poses. Input images for each scene are sampled below each reconstruction.
• Figure 2: We illustrate the sparse representation of Jacobian $\textbf{J}^S$ and sparse-aware operations for fast LM optimization, including sparse matrix-matrix multiplication $\textbf{A}=\textbf{J}^{sT}\textbf{J}^s$, sparse-dense matrix-vector multiplication $\textbf{J}^{sT}\textbf{r}$, and diagonal operations such as element fetching $diag(\textbf{A})$, where both are implemented with zhan2024bundle. White elements are all with zero values.
• Figure 3: Comparisons of total SfM running time among COLMAP, GLOMAP and InstantSfM on different numbers of images. Running time is in log space.
• Figure 4: Comparisons of BA and GP running time between GLOMAP and InstantSfM on different numbers of images to be calibrated. Running time is in log space.
• Figure : The Levenberg-Marquardt algorithm