ZeroGS: Training 3D Gaussian Splatting from Unposed Images

TL;DR

This work proposes ZeroGS to train 3DGS from hundreds of unposed and unordered images, and recovers more accurate camera poses than state-of-the-art pose-free NeRF/3DGS methods, and even renders higher quality images than 3DGS with COLMAP poses.

Abstract

Neural radiance fields (NeRF) and 3D Gaussian Splatting (3DGS) are popular techniques to reconstruct and render photo-realistic images. However, the pre-requisite of running Structure-from-Motion (SfM) to get camera poses limits their completeness. While previous methods can reconstruct from a few unposed images, they are not applicable when images are unordered or densely captured. In this work, we propose ZeroGS to train 3DGS from hundreds of unposed and unordered images. Our method leverages a pretrained foundation model as the neural scene representation. Since the accuracy of the predicted pointmaps does not suffice for accurate image registration and high-fidelity image rendering, we propose to mitigate the issue by initializing and finetuning the pretrained model from a seed image. Images are then progressively registered and added to the training buffer, which is further used to train the model. We also propose to refine the camera poses and pointmaps by minimizing a point-to-camera ray consistency loss across multiple views. Experiments on the LLFF dataset, the MipNeRF360 dataset, and the Tanks-and-Temples dataset show that our method recovers more accurate camera poses than state-of-the-art pose-free NeRF/3DGS methods, and even renders higher quality images than 3DGS with COLMAP poses. Our project page is available at https://aibluefisher.github.io/ZeroGS.

Figures (12)

• Figure 1: Reconstruction results of ZeroGS. Our method reconstructs scenes from hundreds of images without COLMAP poses.
• Figure 2: The training pipeline of our Pose-Free 3D Gaussian Splatting. Our method follows the classical incremental SfM reconstruction pipeline with the key difference that the input is no longer an image but a pair of images in a progressively updated training buffer. The scene regressor network is trained as follows: 1) Use Spann3R DBLP:journals/corr/abs-2408-16061 as the scene regressor network to predict 3D Gaussians $\mathbf{G}_k$ and pointmaps $\mathbf{X}_k$ from a pair of images. 2) Leverage RANSAC and a PnP solver to obtain the initial camera poses based on direct 2D-3D correspondences. 3) Refine the coarse camera poses by minimizing the point-to-ray consistency loss between 3D tracks and camera centers. 4) Rasterize the 3D Gaussians with the refined camera poses to render images. An RGB loss is adopted for back-propagating gradients. 5) After each training epoch, we update the training buffer by registering more images.
• Figure 3: Visualization of camera poses accuracy on the LLFF dataset (Zoom in for best view). Black: pseudo-ground-truth camera poses obtained from COLMAP DBLP:conf/cvpr/SchonbergerF16. Colored: predicted camera poses.
• Figure 4: Visualization of camera poses accuracy on the MipNeRF360 dataset (Zoom in for best view). Black: pseudo-ground-truth camera poses obtained from COLMAP DBLP:conf/cvpr/SchonbergerF16. Colored: predicted camera poses.
• Figure 5: The qualitative results of novel view synthesis on LLFF forward-facing dataset DBLP:journals/tog/MildenhallSCKRN19.