R3GW: Relightable 3D Gaussians for Outdoor Scenes in the Wild

TL;DR

R3GW is a novel method that learns a relightable 3DGS representation of an outdoor scene captured in the wild, using two distinct sets of Gaussians, and synthesizes photorealistic novel views under arbitrary illumination conditions.

Abstract

3D Gaussian Splatting (3DGS) has established itself as a leading technique for 3D reconstruction and novel view synthesis of static scenes, achieving outstanding rendering quality and fast training. However, the method does not explicitly model the scene illumination, making it unsuitable for relighting tasks. Furthermore, 3DGS struggles to reconstruct scenes captured in the wild by unconstrained photo collections featuring changing lighting conditions. In this paper, we present R3GW, a novel method that learns a relightable 3DGS representation of an outdoor scene captured in the wild. Our approach separates the scene into a relightable foreground and a non-reflective background (the sky), using two distinct sets of Gaussians. R3GW models view-dependent lighting effects in the foreground reflections by combining Physically Based Rendering with the 3DGS scene representation in a varying illumination setting. We evaluate our method quantitatively and qualitatively on the NeRF-OSR dataset, offering state-of-the-art performance and enhanced support for physically-based relighting of unconstrained scenes. Our method synthesizes photorealistic novel views under arbitrary illumination conditions. Additionally, our representation of the sky mitigates depth reconstruction artifacts, improving rendering quality at the sky-foreground boundary

Figures (11)

• Figure 1: R3GW reconstructs a relightable 3D Gaussian Splatting representation of an outdoor scene from a collection of unconstrained multiview images. Our method factorizes the scene’s foreground into geometry, material, and illumination, with the lighting represented as an environment map. This enables illumination editing along with novel view synthesis. The sky, as a non-reflective surface, is modeled with a dedicated set of Gaussians that remain independent of scene lighting and material.
• Figure 2: Training pipeline of R3GW. R3GW learns a relightable 3DGS representation of an outdoor scene captured in the wild. The PBR color of the foreground Gaussians, depending on the surface normals and material properties at their positions, as well as on the environment light, enables relighting of the scene's foreground. In contrast, the color of the sky Gaussians is independent of the scene illumination. The rendered image is formed by rasterizing the sky and the foreground Gaussians in a single pass. The Gaussians are regularized so that the sky Gaussians are responsible for the rendering of the sky pixels, while the foreground Gaussians contribute only to the foreground region. After training, the foreground Gaussians can be rendered under novel lighting conditions by supplying the corresponding environment map as input.
• Figure 3: Qualitative comparison of the depth maps predicted by our method and LumiGauss kaleta2025lumigauss for the NeRF-OSR scenes. For each novel view, the ground truth is visualized in Fig. \ref{['fig:qualitativecomp_renders']}. Compared to LumiGauss, R3GW produces more geometrically consistent depth maps with significantly reduced artifacts in the sky and in the distant regions of the foreground.
• Figure 4: Qualitative comparison of rendering quality at the sky-foreground boundary between our method and LumiGauss kaleta2025lumigauss for the NeRF-OSR scenes. For a clear visualization, we render the novel views for both R3GW and LumiGauss using trained illuminations. Our sky Gaussians are rendered using the corresponding trained sky color SH. R3GW yields sharper building outlines with fewer halos as a result of an improved depth reconstruction (see Fig. \ref{['fig:qualitativecomp_depth_maps']}).
• Figure 5: Qualitative comparison of the albedo map and surface normals predicted by our method, NeRF-OSR rudnev2022nerf, and LumiGauss kaleta2025lumigauss for the NeRF-OSR scene st. The ground truth novel view is visualized in Fig. \ref{['fig:qualitativecomp_renders']}. R3GW synthesizes sharper surface normal maps than LumiGauss and NeRF-OSR. Furthermore, our sky Gaussians are not attributed with spurious albedo and surface normals.