RNG: Relightable Neural Gaussians

TL;DR

RNG introduces Relightable Neural Gaussians, a 3D Gaussian Splatting-based framework that enables relighting of objects with both hard surfaces and soft boundaries without relying on predefined shading models. Each Gaussian point carries a latent reflectance feature decoded by an MLP conditioned on view and light directions, and a shadow cue with an accompanying depth refinement network improves shadow fidelity. A two-stage hybrid optimization (forward shading followed by deferred shading with shadow cues) balances geometry accuracy and realistic shadows, yielding faster training (~1.3 hours) and real-time rendering (~60 fps) while outperforming NRHints and GS^3 in shadow quality and detail. This approach widens the practical use of relightable 3D assets for complex materials, offering efficient, flexible relighting under novel viewpoints and lighting with potential extensions to more complex light transport phenomena.

Abstract

3D Gaussian Splatting (3DGS) has shown impressive results for the novel view synthesis task, where lighting is assumed to be fixed. However, creating relightable 3D assets, especially for objects with ill-defined shapes (fur, fabric, etc.), remains a challenging task. The decomposition between light, geometry, and material is ambiguous, especially if either smooth surface assumptions or surfacebased analytical shading models do not apply. We propose Relightable Neural Gaussians (RNG), a novel 3DGS-based framework that enables the relighting of objects with both hard surfaces or soft boundaries, while avoiding assumptions on the shading model. We condition the radiance at each point on both view and light directions. We also introduce a shadow cue, as well as a depth refinement network to improve shadow accuracy. Finally, we propose a hybrid forward-deferred fitting strategy to balance geometry and appearance quality. Our method achieves significantly faster training (1.3 hours) and rendering (60 frames per second) compared to a prior method based on neural radiance fields and produces higher-quality shadows than a concurrent 3DGS-based method. Project page: https://www.whois-jiahui.fun/project_pages/RNG.

Figures (15)

• Figure 1: The overview of RNG. Each Gaussian point in the scene contains an extra latent vector that describes the reflectance. The latent values interpreted by an MLP decoder, conditioned on view and light directions. Training has two stages. In the first stage, we employ forward shading, where we decode all the latent vectors of Gaussian points into colors, followed by the alpha blending. In the second deferred shading stage, we first alpha-blend the neural Gaussian features to get an aggregated feature, and then we feed it to the decoder. We apply shadow mapping to obtain a shadow cue map and use the shadow cue as an extra input for the decoder in the second stage.
• Figure 2: The effect of the depth refinement network. The weighted sum of Gaussian depths is not accurate, resulting in mismatching shadow cues. Therefore, we propose a depth refinement network to correct the depth.
• Figure 3: The illustration of shadow cue computation. First, we splat the Gaussians onto the camera to get depth values. Then, we run the depth refinement network to correct them and locate the shading points $P$. At last, we splat the shading points onto the shadow camera to find the intersections of shadow rays $Q$, and store the distance $\vert PQ \vert$ as the shadow cue.
• Figure 4: Comparison between NRHints zeng2023relighting, GS$^3$bi2024rgs and our method on real/synthetic datasets under point lights. The best/second-best results are marked as bold/italic, respectively. Our method has the lowest LPIPS with the shown images and is also the best or second-best in PSNR and SSIM values. Our method also has better shadow areas than GS$^3$.
• Figure 5: Comparison of relighting results under environment lighting with PSNR values of each image. We compare our relighting results with some previous GS-based methods liang_2024_gsirgao_2023_relightablegs and the ground truth. Our method decomposes the light and materials better and achieves better relighting, as we utilize point-lit images and the neural appearance model.