PhotoReg: Photometrically Registering 3D Gaussian Splatting Models

TL;DR

PhotoReg addresses the challenge of fusing multiple photorealistic Gaussian Splatting models into a single, scalable representation suitable for decentralized robotic mapping. It combines 3D foundation models to obtain coarse, scale-aware alignments and refines them with differentiable photometric optimization, ensuring precise alignment even with low overlap. The approach outperforms traditional baselines like ICP and COLMAP on standard benchmarks and robot-collected data, and extends to multi-model fusion. This yields high-quality, shareable environment representations enabling efficient multi-robot collaboration and real-time visualization.

Abstract

Building accurate representations of the environment is critical for intelligent robots to make decisions during deployment. Advances in photorealistic environment models have enabled robots to develop hyper-realistic reconstructions, which can be used to generate images that are intuitive for human inspection. In particular, the recently introduced \ac{3DGS}, which describes the scene with up to millions of primitive ellipsoids, can be rendered in real time. \ac{3DGS} has rapidly gained prominence. However, a critical unsolved problem persists: how can we fuse multiple \ac{3DGS} into a single coherent model? Solving this problem will enable robot teams to jointly build \ac{3DGS} models of their surroundings. A key insight of this work is to leverage the {duality} between photorealistic reconstructions, which render realistic 2D images from 3D structure, and \emph{3D foundation models}, which predict 3D structure from image pairs. To this end, we develop PhotoReg, a framework to register multiple photorealistic \ac{3DGS} models with 3D foundation models. As \ac{3DGS} models are generally built from monocular camera images, they have \emph{arbitrary scale}. To resolve this, PhotoReg actively enforces scale consistency among the different \ac{3DGS} models by considering depth estimates within these models. Then, the alignment is iteratively refined with fine-grained photometric losses to produce high-quality fused \ac{3DGS} models. We rigorously evaluate PhotoReg on both standard benchmark datasets and our custom-collected datasets, including with two quadruped robots. The code is released at \url{ziweny11.github.io/photoreg}.

Figures (10)

• Figure 1: PhotoReg takes two input 3D Gaussian Splatting models and aligns them into a merged Gaussian Splatting model. Transformation and Scale information between two inputs is obtained through 3D visual foundation models and further refined photometrically.
• Figure 2: DUSt3R workflow: Input are two RGB images, output are corresponding depth maps, confidence maps, and a 3D reconstructed scene with camera poses of input images recovered.
• Figure 3: DINOv2 Workflow: Input images are transformed into feature embeddings, with the first three principal components visualized in RGB values. This figure demonstrates that similar objects are mapped to closely related embeddings, and are invariant to differences in position or angle.
• Figure 4: This workflow starts by rendering images from training poses of two overlapping 3D Gaussians. We then select images where the overlap is detected using a visual foundation model. The initial pose between these images is established based on rendered depth maps and a 3D foundation model. By optimizing the photometric loss, we refine the rigid body transformation and scale ratio between the two 3DGS, achieving an aligned Gaussian model.
• Figure 5: Sequence of transformations: $o_1$ and $o_2$ are the coordinate frames of $G_1$ and $G_2$; $c_1$ and $c_2$ are the camera coordinate frames of $G_1$ and $G_2$. $T^{c1}_{c2}$ is not available as the scale is unknown.