3iGS: Factorised Tensorial Illumination for 3D Gaussian Splatting

TL;DR

Factorised Tensorial Illumination for 3D Gaussian Splatting, or 3iGS, improves upon 3D Gaussian Splatting (3DGS) rendering quality by expressing the outgoing radiance as a function of a local illumination field and Bidirectional Reflectance Distribution Function (BRDF) features.

Abstract

The use of 3D Gaussians as representation of radiance fields has enabled high quality novel view synthesis at real-time rendering speed. However, the choice of optimising the outgoing radiance of each Gaussian independently as spherical harmonics results in unsatisfactory view dependent effects. In response to these limitations, our work, Factorised Tensorial Illumination for 3D Gaussian Splatting, or 3iGS, improves upon 3D Gaussian Splatting (3DGS) rendering quality. Instead of optimising a single outgoing radiance parameter, 3iGS enhances 3DGS view-dependent effects by expressing the outgoing radiance as a function of a local illumination field and Bidirectional Reflectance Distribution Function (BRDF) features. We optimise a continuous incident illumination field through a Tensorial Factorisation representation, while separately fine-tuning the BRDF features of each 3D Gaussian relative to this illumination field. Our methodology significantly enhances the rendering quality of specular view-dependent effects of 3DGS, while maintaining rapid training and rendering speeds.

Figures (5)

• Figure 1: We present test renderings from the "Drums" scene within the blender dataset mildenhall2020nerf, comparing our technique against Gaussian Splatting (3DGS) kerbl20233d and the ground truth (G.T). As the perspective shifts around the scene, the colour of the Floor Tom's top changes from translucent to reflective, showcasing intricate effects that depend on the viewpoint. These effects result from the specular reflection of incoming light and the reflections within the scene from elements like the Cymbals. Contrary to 3DGS, which struggles to capture these complex variations in light reflection, our method, 3iGS, aligns more accurately with the ground truth.
• Figure 2: A visualisation of 3iGS pipeline to render a single Gaussian's colour. We interpolate an incident illumination $L_{i}$ from the factorised tensorial illumination field $\mathcal{G}_l$ using a Gaussian mean $\bm{x}_i$ as input. A neural network $\mathcal{F}$ maps the illumination field $L_{i}$, the Gaussian BRDF features $\rho_i$, and the viewing direction $\omega_o$ to Gaussian's specular colour $c_s$. Following, the diffused colour $c_d$ and specular colour $c_s$ are added linearly to produce the final outgoing radiance field $c$.
• Figure 3: Comparisons of test-set views of real world scenes. 3iGS enhances 3DGS renderings by producing clearer view dependent effects as shown.
• Figure 4: In evaluating test-set views from the Shiny Blender dataset, we compared the performance of 3DGS kerbl20233d, GaussianShader jiang2023gaussianshader, and our work 3iGS. The standard 3DGS method generally yields the least satisfactory renderings, with images often appearing blurry in areas of specular reflection. GaussianShader shows a slight improvement by incorporating the GGX BRDF model, leading to marginally better results in rendering specular regions. In contrast, 3iGS stands out by employing a general rendering function that predicts neural features of illumination field and BRDF instead of relying on physical parameters. This approach allows 3iGS to surpass existing methods significantly, capturing the intricate details within specular highlights with remarkable precision.
• Figure 5: In contrast to 3DGS kerbl20233d and GaussianShader jiang2023gaussianshader, our 3iGS method uniquely identifies both the golden specular highlights and the reflections on the Medium Tom as seen in the plastic surface of the Floor Tom (top row). Our approach successfully captures the detailed specular highlights on every cymbal within the drum setup from the Blender dataset, as presented in mildenhall2020nerf.