LumiGauss: Relightable Gaussian Splatting in the Wild

TL;DR

LumiGauss tackles the challenge of decoupling illumination from geometry in unconstrained photo collections by leveraging 2D Gaussian Splatting to model scenes and environment lighting. It introduces per-Gaussian radiance transfer via spherical harmonics to capture shadows, along with an environment-map prediction module guided by learned embeddings, enabling relighting under novel light conditions and viewpoints. The approach yields high-fidelity reconstructions, realistic shadows, and seamless integration with graphics engines through fast precomputed radiance transfer, validated on the NeRF-OSR dataset with favorable comparisons to baselines. This work advances practical inverse rendering in the wild, offering a scalable pathway to relightable 3D assets from casual imagery.

Abstract

Decoupling lighting from geometry using unconstrained photo collections is notoriously challenging. Solving it would benefit many users as creating complex 3D assets takes days of manual labor. Many previous works have attempted to address this issue, often at the expense of output fidelity, which questions the practicality of such methods. We introduce LumiGauss - a technique that tackles 3D reconstruction of scenes and environmental lighting through 2D Gaussian Splatting. Our approach yields high-quality scene reconstructions and enables realistic lighting synthesis under novel environment maps. We also propose a method for enhancing the quality of shadows, common in outdoor scenes, by exploiting spherical harmonics properties. Our approach facilitates seamless integration with game engines and enables the use of fast precomputed radiance transfer. We validate our method on the NeRF-OSR dataset, demonstrating superior performance over baseline methods. Moreover, LumiGauss can synthesize realistic images for unseen environment maps. Our code: https://github.com/joaxkal/lumigauss.

Figures (11)

• Figure 1: Pipeline -- LumiGauss learns the relightable 2D Gaussian zhang2024gaussian representation from unconstrained photo collection with variable camera parameters and lighting conditions. Each of $k$ Gaussians holds: a normal $\mathbf{n}_k$, albedo $\bm{\rho}_k$, and learnable transfer function $\mathbf{d}_k$. Our contributed method composes the Gaussians in two modes---shadowed and unshadowed. The shadowed model reconstructs additional shadows (see \ref{['fig:teaser']}) on top of the unshadowed model thanks to our proposed use of a radiance transfer function. The Gaussians are splatted kerbl20233dgaussiansplattingzhang2024gaussian to render the output image in a novel view and light.
• Figure 2: Unshadowed $\mathbf{c}$ (\ref{['eq:unshadowed-radiance']}) and shadowed $\tilde{\mathbf{c}}$ (\ref{['eq:shadowed-radiance']}) may give the same output color if a Gaussian is fully exposed to the environment light. In the case of any occluder, $\mathbf{c}$ does not handle, and the color does not change. However, our proposed $\tilde{\mathbf{c}}$ properly reacts to the occluder and makes the output color darker.
• Figure 3: Scene reconstruction and relightning -- Reconstruction and relighting capabilities of LumiGauss. LumiGauss reproduces sharp and clean landmarks, and the learned environment lighting enables accurate scene relighting. We use learned environment maps to relight the scene from novel viewpoints and then relight arbitrary objects within a graphics engine.
• Figure 4: Qualitative comparison of albedo, normals, and relighting under similar lighting conditions on Trevi Fountain. LumiGauss produces albedo with fewer baked-in shadows, sharp normals, smooth surfaces, and more accurate novel lighting compared to the baselines. Results for NeuSky originally reported in gardner2023neusky. Please, zoom in for details.
• Figure 5: Effects of shadowed training -- We show the comparison of albedo between the shadowed (left) and unshadowed (right) models. The albedo in the shadowed training is brighter with fewer shadows. The shadowed model recovers more accurate normals.