SVG-IR: Spatially-Varying Gaussian Splatting for Inverse Rendering

TL;DR

SVG-IR introduces a Spatially-varying Gaussian representation and a dedicated SVG splatting render pipeline to enable per-Gaussian spatially varying materials and normals for inverse rendering. By coupling this with a physically based indirect illumination model and one-bounce relighting via secondary Gaussians, the approach decouples lighting and material more effectively and produces realistic indirect lighting under novel environments. Empirically, SVG-IR outperforms state-of-the-art NeRF-based IR by about 2.5 dB in PSNR and Gaussian-based relighting methods by about 3.5 dB, while maintaining real-time rendering. The work highlights the benefits of interpolated shading, differentiable rasterization analogies, and physically grounded light transport for robust NVS and relighting.

Abstract

Reconstructing 3D assets from images, known as inverse rendering (IR), remains a challenging task due to its ill-posed nature. 3D Gaussian Splatting (3DGS) has demonstrated impressive capabilities for novel view synthesis (NVS) tasks. Methods apply it to relighting by separating radiance into BRDF parameters and lighting, yet produce inferior relighting quality with artifacts and unnatural indirect illumination due to the limited capability of each Gaussian, which has constant material parameters and normal, alongside the absence of physical constraints for indirect lighting. In this paper, we present a novel framework called Spatially-vayring Gaussian Inverse Rendering (SVG-IR), aimed at enhancing both NVS and relighting quality. To this end, we propose a new representation-Spatially-varying Gaussian (SVG)-that allows per-Gaussian spatially varying parameters. This enhanced representation is complemented by a SVG splatting scheme akin to vertex/fragment shading in traditional graphics pipelines. Furthermore, we integrate a physically-based indirect lighting model, enabling more realistic relighting. The proposed SVG-IR framework significantly improves rendering quality, outperforming state-of-the-art NeRF-based methods by 2.5 dB in peak signal-to-noise ratio (PSNR) and surpassing existing Gaussian-based techniques by 3.5 dB in relighting tasks, all while maintaining a real-time rendering speed.

Figures (10)

• Figure 1: SVG-IR introduces a new spatially-varying Gaussian representation inspired by replacing flat shading for interpolated shading in triangle rendering. Each spatially-varying Gaussian allows spatially-varying material and normal attributes by interpolating among Gaussian vertices defined in the tangent space of a Gaussian. Compared to the original Gaussian (i.e., Constant Gaussian) with constant attributes, SVG has a more powerful representation ability to produce high-quality NVS and relighting results.
• Figure 2: Illustration of two Constant Gaussians and Spatially-varying Gaussians fitting a distribution of BRDFs. As Spatially-varying Gaussians allow spatially-varying parameters, a single Gaussian can have different BRDF lobes at different places, leading to a more flexible representation compared to the Constant Gaussian.
• Figure 3: Overview of our framework. We propose a novel SVG-IR framework. Within this framework, we introduce a Spatially-varying Gaussian representation capable of spatially variability with material attributes. We employ SVG splatting, analogous to vertex and fragment shading in the traditional triangle rendering pipeline, to leverage the improved appearance capability of SVGs. Additionally, we present a physics-based lighting model that enforces additional physical constraints to facilitate the decoupling of lighting and material properties.
• Figure 4: The results of rendering using only indirect lighting. We multiplied the brightness of the right half by two for better viewing. GS-IR fails to capture the effects of light reflections from objects as it only models brightness. Meanwhile, Relightable 3DGS lacks supervision for indirect lighting, leading to unnatural results. In contrast, our physically-based lighting model produces more natural rendering results.
• Figure 5: Illustration of the physically-based illumination and our one-bounce relighting method when relighting.