A Taxonomy of Structure from Motion Methods

TL;DR

This paper presents a conceptual taxonomy for Structure from Motion (SfM) methods by partitioning approaches into Structure and Motion (SAM), Structure from Motion (SFM), and Structure without Motion (SWM). It emphasizes graph-based formulations and analyzes theoretical conditions that ensure well-posedness, including degeneracies and solvability criteria for calibrated and uncalibrated cameras. The survey covers classic geometry-based pipelines and also surveys recent data-driven trends, including learning-based matrix completion and differentiable SfM components. It highlights the trade-offs between sequential/global strategies, initialization requirements, and the potential for combining theory with practice to handle degeneracies and large-scale data. The work aims to guide researchers toward principled SfM designs with clear assumptions and provable properties.

Abstract

Structure from Motion (SfM) refers to the problem of recovering both structure (i.e., 3D coordinates of points in the scene) and motion (i.e., camera matrices) starting from point correspondences in multiple images. It has attracted significant attention over the years, counting practical reconstruction pipelines as well as theoretical results. This paper is conceived as a conceptual review of SfM methods, which are grouped into three main categories, according to which part of the problem - between motion and structure - they focus on. The proposed taxonomy brings a new perspective on existing SfM approaches as well as insights into open problems and possible future research directions. Particular emphasis is given on identifying the theoretical conditions that make SfM well posed, which depend on the problem formulation that is being considered.

Figures (12)

• Figure 1: The goal of Structure from Motion (SfM) is to recover both camera motion and scene structure starting from point correspondences in multiple images. According to the proposed taxonomy, methods can be divided into three categories, based on which part of the problem they focus on (motion, structure, motion and structure). Each category can be further divided into sub-categories based on the approach/assumptions, as described in the text.
• Figure 2: The first category in the proposed SfM taxonomy recovers structure and motion simultaneously, starting from point correspondences. A final refinement (bundle adjustment) is applied at the end. Sequential/Hierarchical SfM and the projecive factorization belong to this category.
• Figure 3: Sequential SfM belongs to the first category in the proposed taxonomy, which considers structure and motion in a joint manner. Assuming that images are organized in a sequence, two-view reconstruction is first performed; then, the remaining images are progressively included by alternating resection and intersection, thereby estimating new cameras and 3D points.
• Figure 4: There exist several ways to organize images in convenient abstract structures. In a sequence, images are consecutively captured, as happens (e.g.) in a video. In a tree, there are hierarchical relationships between overlapping images, where each node/image is connected with the parent node. In a viewing graph, each node corresponds to an image and overlapping images are put in relation via edges -- this is suitable for unordered image collections (e.g., taken from the internet). Note that we can view the sequence and the tree as special cases of the viewing graph, which is the most general representation.
• Figure 5: Projective factorization belongs to the first category in the proposed taxonomy, which considers structure and motion in a joint manner. The idea is to collect all the correspondences in a single matrix, which is then factorized into a motion matrix and a structure matrix, under suitable assumptions.

Theorems & Definitions (15)

• Remark 1
• Remark 2
• Remark 3
• Remark 4
• Remark 5
• Remark 6
• Remark 7
• Remark 8
• Remark 9
• Remark 10