Scene Coordinate Reconstruction Priors

TL;DR

This work reframes SCR training as a probabilistic likelihood problem and introduces reconstruction priors to combat degeneracy when multi-view constraints are weak. It proposes a depth distribution prior for RGB data and a learned 3D point-cloud diffusion prior trained on indoor scans, both integrated into ACE/ACE0/GLACE without adding test-time cost. Experiments on indoor datasets show improved scene coherence, higher registration rates, and better camera poses, with positive downstream effects on novel view synthesis and relocalization. The approach demonstrates robustness across multiple SCR variants and highlights potential for RGB-D extensions, while noting limitations for outdoor scenes and suggesting directions for richer priors and conditioning signals.

Abstract

Scene coordinate regression (SCR) models have proven to be powerful implicit scene representations for 3D vision, enabling visual relocalization and structure-from-motion. SCR models are trained specifically for one scene. If training images imply insufficient multi-view constraints SCR models degenerate. We present a probabilistic reinterpretation of training SCR models, which allows us to infuse high-level reconstruction priors. We investigate multiple such priors, ranging from simple priors over the distribution of reconstructed depth values to learned priors over plausible scene coordinate configurations. For the latter, we train a 3D point cloud diffusion model on a large corpus of indoor scans. Our priors push predicted 3D scene points towards plausible geometry at each training step to increase their likelihood. On three indoor datasets our priors help learning better scene representations, resulting in more coherent scene point clouds, higher registration rates and better camera poses, with a positive effect on down-stream tasks such as novel view synthesis and camera relocalization.

Figures (8)

• Figure 1: Reconstruction Priors. ACE0 brachmann2024acezero, a neural SfM pipeline, struggles with an indoor scene (left). The reconstruction partly degenerates, as is evident by the dispersed point cloud, camera poses (green) penetrating a wall, and artifacts in one of the views synthesized with Splatfacto nerfstudio (bottom). We incorporate various priors into ACE0 which lead to a more consistent scene layout, better pose estimates, and fewer artifacts in synthesized views (center). We also show a version of ACE0 taking RGB-D rather than RGB images as input (right).
• Figure 2: System. Left: We train a scene coordinate regression (SCR) model to represent a scene. Following ACE brachmann2023ace and ACE0 brachmann2024acezero, the SCR network predicts scene coordinates for a batch of random image patches in each training iteration. We supervise the SCR network with the reprojection error of scene coordinates w.r.t. ground truth camera poses. Right: Extending brachmann2023acebrachmann2024acezero, we introduce a reconstruction prior that outputs the gradient of the log-likelihood of the predicted point cloud. The prior guides the SCR model to learn a scene representation that is more likely to correspond to a real scene. We investigate various priors: A depth distribution prior encourages depth values of reconstructed scene coordinates to follow a plausible distribution. A 3D point cloud diffusion prior is a network that was trained offline on a large corpus of scenes and encodes plausible scene layouts. We can also infuse a depth prior if inputs are RGB-D.
• Figure 3: Diffusion Prior on Stairs. We guide the training of ACE brachmann2023ace on the Stairs scene with our 3D diffusion model, leading to a more coherent scene geometry, and higher pose estimation accuracy on test images (5cm,5$^\circ$ threshold).
• Figure 4: Generated Point Clouds. Point clouds generated by our diffusion model together with ScanNet point clouds used for training.
• Figure 5: Diffusion Process vs. ACE Training. The evolution of point clouds during ACE training does not match the forward diffusion process over the full range. Hence, we align the diffusion time steps 10-0 with the ACE iterations 5,000-25,000.