DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion

TL;DR

DiffusionSfM tackles the joint problem of recovering 3D structure and camera motion from sparse multi-view imagery by predicting dense per-pixel ray origins and endpoints in a global frame using a denoising diffusion transformer. It unifies geometry and pose estimation into a single end-to-end framework, leveraging GT mask conditioning and a homogeneous representation to robustly learn from incomplete data and unbounded coordinates. Empirically, it achieves superior camera center accuracy and competitive geometry metrics on CO3D and scene-level datasets, while providing natural uncertainty modeling and the ability to generate multi-modal predictions. The work offers a practical, scalable alternative to traditional SfM pipelines and points to future enhancements via latent-space representations and efficiency-driven refinements for larger view settings.

Abstract

Current Structure-from-Motion (SfM) methods typically follow a two-stage pipeline, combining learned or geometric pairwise reasoning with a subsequent global optimization step. In contrast, we propose a data-driven multi-view reasoning approach that directly infers 3D scene geometry and camera poses from multi-view images. Our framework, DiffusionSfM, parameterizes scene geometry and cameras as pixel-wise ray origins and endpoints in a global frame and employs a transformer-based denoising diffusion model to predict them from multi-view inputs. To address practical challenges in training diffusion models with missing data and unbounded scene coordinates, we introduce specialized mechanisms that ensure robust learning. We empirically validate DiffusionSfM on both synthetic and real datasets, demonstrating that it outperforms classical and learning-based approaches while naturally modeling uncertainty.

Figures (10)

• Figure 1: DiffusionSfM.Top: Given a set of multi-view images (left), DiffusionSfM represents scene geometry and cameras (right) as pixel-wise ray origins and endpoints in a global frame. It learns a denoising diffusion model to infer these elements directly from multi-view inputs. Unlike traditional Structure-from-Motion (SfM) pipelines, which separate pairwise reasoning and global optimization into two stages, our approach unifies both into a single end-to-end multi-view reasoning framework. Bottom: Example results of inferred scene geometry and cameras for two distinct settings: a real-world outdoor scene (left) and a synthetic indoor scene (right).
• Figure 2: Method. Given sparse multi-view images as input, DiffusionSfM predicts pixel-wise ray origins and endpoints in a global frame (Sec. \ref{['sec:origins_and_endpoints']}) using a denoising diffusion process (Sec. \ref{['sec:architecture']}). For each image, we compute patch-wise embeddings with DINOv2 oquab2023dinov2 and embed noisy ray origins and endpoints into latents using a single downsampling convolutional layer, ensuring alignment with the spatial footprint of the image embeddings. We implement a Diffusion Transformer architecture that predicts clean ray origins and endpoints from noisy samples. A convolutional DPT ranftl2021vision head outputs full-resolution denoised ray origins and endpoints. To handle incomplete ground truth (GT) during training, we condition the model on GT masks (Sec. \ref{['sec:training_details']}). At inference, the GT masks are set to all ones, enabling the model to predict origins and endpoints for all pixels. The predicted ray origins and endpoints can be directly visualized in 3D or post-processed to recover camera extrinsics, intrinsics, and multi-view consistent depth maps.
• Figure 3: Qualitative Comparison on Camera Pose Accuracy and Predicted Geometry. For each method, we plot the ground-truth cameras in black and the predicted cameras in other colors. DiffusionSfM demonstrates robust performance even with challenging inputs. Compared to DUSt3R, which sometimes fails to register images in a consistent manner, DiffusionSfM consistently yields a coherent global prediction. Additionally, while we observe that DUSt3R can predict highly precise camera rotations, it often struggles with camera centers (see the backpack example). Input images depicting scenes are out-of-distribution for RayDiffusion, as it is trained on CO3D only.
• Figure 4: Additional Qualitative Results on Predicted Camera Poses. DiffusionSfM shows robustness to ambiguous patterns in inputs.
• Figure 5: Multi-modality of DiffusionSfM. We show two distinct samples from DiffusionSfM, starting from the same input images but with different random noise. Sample 1 explains the input images by putting all flowers on the left side, while Sample 2 places one flower on each side (note the difference in the green camera's viewpoint). DiffusionSfM is able to predict multi-modal geometry distributions when the scene layout is ambiguous in the inputs.