Normal-GS: 3D Gaussian Splatting with Normal-Involved Rendering

TL;DR

Normal-GS is introduced, a novel approach that integrates normal vectors into the 3DGS rendering pipeline and model the interaction between normals and incident lighting using the physically-based rendering equation, achieving near state-of-the-art visual quality while obtaining accurate surface normals and preserving real-time rendering performance.

Abstract

Rendering and reconstruction are long-standing topics in computer vision and graphics. Achieving both high rendering quality and accurate geometry is a challenge. Recent advancements in 3D Gaussian Splatting (3DGS) have enabled high-fidelity novel view synthesis at real-time speeds. However, the noisy and discrete nature of 3D Gaussian primitives hinders accurate surface estimation. Previous attempts to regularize 3D Gaussian normals often degrade rendering quality due to the fundamental disconnect between normal vectors and the rendering pipeline in 3DGS-based methods. Therefore, we introduce Normal-GS, a novel approach that integrates normal vectors into the 3DGS rendering pipeline. The core idea is to model the interaction between normals and incident lighting using the physically-based rendering equation. Our approach re-parameterizes surface colors as the product of normals and a designed Integrated Directional Illumination Vector (IDIV). To optimize memory usage and simplify optimization, we employ an anchor-based 3DGS to implicitly encode locally-shared IDIVs. Additionally, Normal-GS leverages optimized normals and Integrated Directional Encoding (IDE) to accurately model specular effects, enhancing both rendering quality and surface normal precision. Extensive experiments demonstrate that Normal-GS achieves near state-of-the-art visual quality while obtaining accurate surface normals and preserving real-time rendering performance.

Figures (5)

• Figure 1: The "seesaw" characteristic between the rendering quality and the normal accuracy of 3DGS-based methods. Our Normal-GS is able to efficiently achieve accurate normal estimation while preserving competitive rendering quality. Our method successfully captures the normals of the cover of the semi-transparent box behind the table.
• Figure 2: Our normal-involved GS, Normal-GS, reparameterizes the original colors into the diffuse and specular components, $c=L_\textrm{D} + L_S$ (bottom). It models the diffuse component as the dot product between the normal vector $\boldsymbol{n}$ and the Integrated Directional Illumination Vector (IDIV) $\boldsymbol{l}$, and utilizes the Integrated Directional Encoding (IDE) verbin2022ref to capture view-dependent specular effects. Inherent parameters are encoded implicitly by a locally shared anchor Gaussian (left) and decoded using MLPs (top). Our method accounts for the contributions of normals to colors, effectively enhancing geometry accuracy and rendering quality.
• Figure 3: Qualitative comparisons of the rendering quality and normal estimation. Our method produced clean normals estimation and preserved good rendering quality.
• Figure 4: b) Our method successfully captures specular effects. c) GShader jiang2023gaussianshader relying on an environment map performs poorly when d) its learned environment map fails. Scenes are taken from the Mip-NeRF 360 dataset barron2022mipnerf360, which have complicated lighting conditions.
• Figure 5: Ablation study about our proposed components. The IDIV together with our regularization strategy produces better quality in rendering quality and normal accuracy.