PRTGaussian: Efficient Relighting Using 3D Gaussians with Precomputed Radiance Transfer

TL;DR

PRTGaussian tackles real-time relighting of general objects from multi-view OLAT data by marrying explicit 3D Gaussian Splatting with Precomputed Radiance Transfer. The method employs a two-stage training pipeline: Stage 1 reconstructs a coarse geometry from multi-view data using vanilla 3D Gaussian Splatting to initialize Gaussian positions, and Stage 2 refines the Gaussians while learning per-Gaussian radiance transfer via a position-encoded neural network, using high-order spherical harmonics (up to order $n=9$) to capture lighting effects. Quantitative and qualitative results on synthetic data show significant speedups in training and rendering with competitive or superior relighting quality compared to state-of-the-art OLAT-based methods, and code is publicly available. This approach enables efficient, free-viewpoint relighting suitable for interactive applications like VR and AR, while recognizing limitations to diffuse materials and low-frequency SH-based lighting.

Abstract

We present PRTGaussian, a realtime relightable novel-view synthesis method made possible by combining 3D Gaussians and Precomputed Radiance Transfer (PRT). By fitting relightable Gaussians to multi-view OLAT data, our method enables real-time, free-viewpoint relighting. By estimating the radiance transfer based on high-order spherical harmonics, we achieve a balance between capturing detailed relighting effects and maintaining computational efficiency. We utilize a two-stage process: in the first stage, we reconstruct a coarse geometry of the object from multi-view images. In the second stage, we initialize 3D Gaussians with the obtained point cloud, then simultaneously refine the coarse geometry and learn the light transport for each Gaussian. Extensive experiments on synthetic datasets show that our approach can achieve fast and high-quality relighting for general objects. Code and data are available at https://github.com/zhanglbthu/PRTGaussian.

Figures (7)

• Figure 1: Overview of Pipeline. Given a multi-view OLAT dataset, our method first reconstruct the initial point cloud as the initialization for the 3D Gaussians. In the next stage, the color of each Gaussian can be obtained by combining the learned radiance transfer, albedo and the light source approximated by spherical harmonics (SH). We jointly optimize the attributes of the Gaussians as well as the encoder and decoder by minimizing the loss between rendered images and ground truth images.
• Figure 2: Qualitative Comparisons. We performed qualitative comparisons with the state-of-the-art method NRHintsnrhints. Here we present the ground truth and the rendering results of different methods for three subjects under two novel lighting and view conditions.
• Figure 3: Data Synthesis Setup. Left represents the setup in Blender and Right represents the sampled positions of the cameras and light sources.
• Figure 4: Qualitative Results. The top shows different environment maps, while the bottom displays the corresponding results using our model trained on the OLAT dataset.
• Figure 5: Ablation Study: The order of the SH basis. Compared to the ground truth (a), using higher-order SH coefficients can enhance the sharpness of shadows. To balance quality and efficiency, we choose $n=9$ in our method.