R3GS: Gaussian Splatting for Robust Reconstruction and Relocalization in Unconstrained Image Collections

TL;DR

R3GS tackles robust reconstruction and relocalization from unconstrained image collections by marrying the efficiency of explicit 3D Gaussian Splatting with a hybrid appearance model that fuses global CNN features and local hash-grid cues. It introduces a transient-object visibility map learned via fine-tuning a lightweight detector, a depth-prior sky handling strategy on a sky sphere, and a relocalization pipeline that is robust to lighting variations. Empirical results on in-the-wild Phototourism datasets show state-of-the-art neural view synthesis performance and improved relocalization accuracy, while achieving faster training and rendering with lower memory compared to prior 3DGS methods. The approach also supports exporting standard 3DGS outputs and will release code open-source after acceptance, enabling practical adoption and further research in outdoor scene reconstruction and relocalization.

Abstract

We propose R3GS, a robust reconstruction and relocalization framework tailored for unconstrained datasets. Our method uses a hybrid representation during training. Each anchor combines a global feature from a convolutional neural network (CNN) with a local feature encoded by the multiresolution hash grids [2]. Subsequently, several shallow multi-layer perceptrons (MLPs) predict the attributes of each Gaussians, including color, opacity, and covariance. To mitigate the adverse effects of transient objects on the reconstruction process, we ffne-tune a lightweight human detection network. Once ffne-tuned, this network generates a visibility map that efffciently generalizes to other transient objects (such as posters, banners, and cars) with minimal need for further adaptation. Additionally, to address the challenges posed by sky regions in outdoor scenes, we propose an effective sky-handling technique that incorporates a depth prior as a constraint. This allows the inffnitely distant sky to be represented on the surface of a large-radius sky sphere, signiffcantly reducing ffoaters caused by errors in sky reconstruction. Furthermore, we introduce a novel relocalization method that remains robust to changes in lighting conditions while estimating the camera pose of a given image within the reconstructed 3DGS scene. As a result, R3GS significantly enhances rendering ffdelity, improves both training and rendering efffciency, and reduces storage requirements. Our method achieves state-of-the-art performance compared to baseline methods on in-the-wild datasets. The code will be made open-source following the acceptance of the paper.

Figures (3)

• Figure 1: Given an unconstrained image collection as input, R3GS achieves high relocalization accuracy through appearance tuning, significantly outperforming 3DGS in terms of Absolute Trajectory Error (ATE).
• Figure 3: Visual comparison of rendering quality between different approaches. Red and green crops emphasize geometry differences.
• Figure 4: Ablation study results: (a) Ground truth; (b) rendered result; (c) rendered result without the visibility map; (d) rendered result without sky handling; (f) visibility map; (g) rendered depth; (h) rendered depth without sky handling. Notably, in the absence of the visibility map, the rendered result exhibits more artifacts on the floor. Additionally, without sky handling, the depth of the sky overlaps with the foreground, leading to inconsistencies.