PRTGS: Precomputed Radiance Transfer of Gaussian Splats for Real-Time High-Quality Relighting

TL;DR

PRTGS tackles real-time, high-quality relighting for 3D Gaussian splats under low-frequency lighting by precomputing radiance transfer functions per splat and encoding them in SH-domain representations. The method introduces distinct precompute strategies for training and rendering and a one-bounce Gaussian ray tracing pipeline to estimate indirect illumination efficiently. Experimental results on synthetic and real-world datasets show state-of-the-art visual quality among real-time methods and substantial training-time speedups compared with TensoIR, while maintaining 1080p 30+ fps relighting. The work enables accurate soft shadows, interreflections, and scene editing in dynamic lighting scenarios, with practical implications for real-time graphics and interactive multimedia.

Abstract

We proposed Precomputed RadianceTransfer of GaussianSplats (PRTGS), a real-time high-quality relighting method for Gaussian splats in low-frequency lighting environments that captures soft shadows and interreflections by precomputing 3D Gaussian splats' radiance transfer. Existing studies have demonstrated that 3D Gaussian splatting (3DGS) outperforms neural fields' efficiency for dynamic lighting scenarios. However, the current relighting method based on 3DGS still struggles to compute high-quality shadow and indirect illumination in real time for dynamic light, leading to unrealistic rendering results. We solve this problem by precomputing the expensive transport simulations required for complex transfer functions like shadowing, the resulting transfer functions are represented as dense sets of vectors or matrices for every Gaussian splat. We introduce distinct precomputing methods tailored for training and rendering stages, along with unique ray tracing and indirect lighting precomputation techniques for 3D Gaussian splats to accelerate training speed and compute accurate indirect lighting related to environment light. Experimental analyses demonstrate that our approach achieves state-of-the-art visual quality while maintaining competitive training times and allows high-quality real-time (30+ fps) relighting for dynamic light and relatively complex scenes at 1080p resolution.

Figures (4)

• Figure 1: The proposed rendering pipeline. Starting with a collection of 3D Gaussian splats that embody geometry, visibility, and BRDF attributes along with incident lighting, we first compute the radiance transfer for every Gaussian splat by executing equation \ref{['glossy transfer']}, the direct transfer matrix by equation \ref{['glossy prt']} and the incident vector by equation \ref{['sh2']}. Following this, we conduct one-bounce 3D Gaussian ray tracing (Sec 4.4) to get the index matrix and precompute self-transfer for every Gaussian splat recursively to estimate indirect illumination based on the index matrix. Finally, we conduct a straightforward dot product between the radiance transfer matrix and incident vector in the spherical harmonics domain and compute the final rendering result with Gaussian splatting.
• Figure 2: 3D Gaussian raytracing. Ray from $G_1$ at direction 1 hits 3 Gaussian splats $G_2$, $G_3$ and $G4$, we compute the weight for each Gaussian splat by equation \ref{['weight']} and select the Gaussian splat with the biggest weight ($G_2$).
• Figure 3: High-quality relighting results achieved by our proposed method on TensoIR dataset jin2023tensoir, DTU dataset jensen2014large and a composite scene created by us.
• Figure 4: Qualitative comparison on TensoIR Synthetic dataset. We visualize the indirect illumination in different lighting conditions. To showcase more details, we have merged the doubled-scaled indirect illumination results (on the right half) with the original brightness indirect illumination results (on the left half). The top-left corner exhibits a magnified view of the local area.