GS^3: Efficient Relighting with Triple Gaussian Splatting

TL;DR

A spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images, and results compare favorably with state-of-the-art techniques in terms of quality/performance.

Abstract

We present a spatial and angular Gaussian based representation and a triple splatting process, for real-time, high-quality novel lighting-and-view synthesis from multi-view point-lit input images. To describe complex appearance, we employ a Lambertian plus a mixture of angular Gaussians as an effective reflectance function for each spatial Gaussian. To generate self-shadow, we splat all spatial Gaussians towards the light source to obtain shadow values, which are further refined by a small multi-layer perceptron. To compensate for other effects like global illumination, another network is trained to compute and add a per-spatial-Gaussian RGB tuple. The effectiveness of our representation is demonstrated on 30 samples with a wide variation in geometry (from solid to fluffy) and appearance (from translucent to anisotropic), as well as using different forms of input data, including rendered images of synthetic/reconstructed objects, photographs captured with a handheld camera and a flash, or from a professional lightstage. We achieve a training time of 40-70 minutes and a rendering speed of 90 fps on a single commodity GPU. Our results compare favorably with state-of-the-art techniques in terms of quality/performance. Our code and data are publicly available at https://GSrelight.github.io/.

Figures (11)

• Figure 1: Our deferred-shading-based pipeline. First, we color each spatial Gaussian by evaluating its appearance function, defined as a Lambertian plus a linear combination of basis angular Gaussians, and splat into a shading image. Next, for each spatial Gaussian, we compute a shadow value by splatting all of them towards the light (i.e., shadow splatting), and refine it with an MLP. We color each spatial Gaussian with its shadow value, and splat them into a shadow image. Finally, we color each spatial Gaussian with another MLP that represents unhandled effects like global illumination, and splat them into a residual image. The final rendering result is computed by multiplying the shading image with the shadow one, and adding the residual image, on a per-pixel basis.
• Figure 2: Shadow computation. We first splat all spatial Gaussians towards the light source, and compute the accumulated opacity $T_m$ and the density ${\beta}_m$ at each intersection with each shadow ray (left). For each spatial Gaussian, all of its ${T}_m$ with respect to different shadow rays are weighted-averaged by corresponding ${\beta}_m$ to obtain a shadow value $T$ (center). Next, this shadow value is refined with a small MLP. Finally, Gaussians with refined shadow values ($T'$) are splatted towards the camera to produce the shadow image (right). The above Gaussians may appear incorrectly, when viewed in a program other than Adobe Acrobat.
• Figure 3: Visualization of the basis angular Gaussians and the spatial distributions of corresponding weights. Each image shows the spatial distribution of the weight for a particular basis angular Gaussian, which is color-coded in the bottom-left corner. A brighter pixel indicates a larger weight.
• Figure 4: Our relighting results on captured data from a professional lightstage. For each pair of images, the left one is the ground-truth photograph, and the right is our result. Average errors in SSIM, PSNR and LPIPS are reported at the bottom.
• Figure 5: Our relighting results on captured data from zeng2023nrhints. For each pair of images, the left one is the ground-truth photograph, and the right is our rendering result. Average errors in SSIM, PSNR and LPIPS are reported at the bottom.