Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing

TL;DR

This work extends explicit 3D Gaussian Splatting to relightable scenes by assigning per-point normals, BRDF parameters, and incident lighting to 3D Gaussians. It uses a physically based rendering equation with a globally shared environment map and per-Gaussian indirect lighting, combined with a BVH-based point ray tracer to produce realistic shadows. The optimization is performed in two stages: first recovering geometry and normals, then fixing geometry while learning materials and lighting, with several regularizations to stabilize decomposition. Experiments on NeRF-synthetic, Synthetic4Relight, and Mip-NeRF 360 demonstrate improved BRDF estimation, novel-view synthesis quality, and convincing relighting, including multi-object scenes. The framework supports editing and relighting entirely on a point-based system, offering a scalable alternative to mesh-based pipelines.

Abstract

In this paper, we present a novel differentiable point-based rendering framework to achieve photo-realistic relighting. To make the reconstructed scene relightable, we enhance vanilla 3D Gaussians by associating extra properties, including normal vectors, BRDF parameters, and incident lighting from various directions. From a collection of multi-view images, the 3D scene is optimized through 3D Gaussian Splatting while BRDF and lighting are decomposed by physically based differentiable rendering. To produce plausible shadow effects in photo-realistic relighting, we introduce an innovative point-based ray tracing with the bounding volume hierarchies for efficient visibility pre-computation. Extensive experiments demonstrate our improved BRDF estimation, novel view synthesis and relighting results compared to state-of-the-art approaches. The proposed framework showcases the potential to revolutionize the mesh-based graphics pipeline with a point-based pipeline enabling editing, tracing, and relighting.

Figures (7)

• Figure 1: Visual results of our pipeline on a multi-object composition scene. In our pipeline, we represent a scene as Relightable 3D Gaussians. From multi-view images, we recover the geometry and materials of individual objects with inverse rendering techniques (see Sec. \ref{['sec:optimization']}). Then, objects are easily composited into a new scene, thanks to our explicit representation. After that, we solve the complex occlusions though point based ray tracing (see Sec. \ref{['sec:raytracing']}) and re-light the new scene. Ultimately, we achieve high-fidelity relighting with remarkably realistic shadow.
• Figure 2: The proposed differentiable rendering pipeline. Starting with a collection of 3D Gaussians that embody geometry, materials, and lighting attributes, we first execute the physically based rendering equation for each 3D Gaussian to determine its outgoing radiance, denoted as $\boldsymbol{c'}$. Following this, we perform rasterization and alpha blending to obtain vanilla color map $\mathcal{C}$, PBR color map $\mathcal{C'}$, depth map $\mathcal{D}$, normal map $\mathcal{N}$, etc. To optimize relightable 3D Gaussians, we utilize the ground truth image $\mathcal{C}_{gt}$ and the pseudo normal map derived from $\mathcal{D}$ for supervision.
• Figure 3: Intersection tests in point-based ray tracing. Intersection point between ray and Gaussian is obtained by three steps: (a) intersect the BVH root node; (b) dive into the intersected child nodes recursively until the leaf node; (c) perform Eq. \ref{['eq:intersect']} to get the equivalent intersection point. (d) shows that a 3D Gaussian actually has non-negligible effect on a segment of a ray.
• Figure 4: Visualizations on NeRF synthetic datasetmildenhall2020nerf. Each scene is displayed in an order from left to right and from top to bottom: Ground Truth, PBR Image, Normal Map and Ambient Occlusion Map.
• Figure 5: Qualitative results on Synthetic4Relight dataset zhang2022modeling.