Generalizable and Relightable Gaussian Splatting for Human Novel View Synthesis

TL;DR

This work introduces GRGS, a generalizable and relightable 3D Gaussian framework for human novel view synthesis that operates without per-character optimization. It combines Lighting-robust Geometry Refinement (LGR) to recover robust depth and high-quality normals with a Physically Grounded Neural Rendering (PGNR) module that fuses physics-based shading and neural prediction, aided by a 2D-to-3D projection training strategy to minimize explicit ray tracing. The method parameterizes and predicts geometry-aware Gaussian attributes and SH-based light transport, enabling editable relighting with shadows and indirect illumination. Experiments on multi-view human datasets and real-world data demonstrate improved geometric fidelity, relighting realism, and strong generalization across identities and illumination conditions, with efficient inference around 20 FPS on a high-end GPU.

Abstract

We propose GRGS, a generalizable and relightable 3D Gaussian framework for high-fidelity human novel view synthesis under diverse lighting conditions. Unlike existing methods that rely on per-character optimization or ignore physical constraints, GRGS adopts a feed-forward, fully supervised strategy projecting geometry, material, and illumination cues from multi-view 2D observations into 3D Gaussian representations. To recover accurate geometry under diverse lighting conditions, we introduce a Lighting-robust Geometry Refinement (LGR) module trained on synthetically relit data to predict precise depth and surface normals. Based on the high-quality geometry, a Physically Grounded Neural Rendering (PGNR) module is further proposed to integrate neural prediction with physics-based shading, supporting editable relighting with shadows and indirect illumination. Moreover, we design a 2D-to-3D projection training scheme leveraging differentiable supervision from ambient occlusion, direct, and indirect lighting maps, alleviating the computational cost of ray tracing. Extensive experiments demonstrate that GRGS achieves superior visual quality, geometric consistency, and generalization across characters and lighting conditions.

Figures (14)

• Figure 1: Given multi-view inputs and environment maps, GRGS reconstructs generalizable 3D representations with high-quality geometry and material while supporting realistic relighting rendering from arbitrary viewpoints.
• Figure 2: Overview of GRGS. Given sparse-view images of a performer under arbitrary illumination, GRGS first leverages the LGR module to reconstruct accurate depth and surface normals, and then employs the PGNR module for material decomposition and physically plausible realistic relighting from novel viewpoints.
• Figure 3: Geometry consistency comparison between GRGS and GPS-Gaussian under varying illumination. We visualize the normal difference heatmap, where cold colors indicate small angular deviations and hot colors denote large ones. GPS-Gaussian exhibits noticeable degradation under extreme lighting.
• Figure 4: Qualitative comparison of our method and 3DGS-based methods. Zoom in for the best view.
• Figure 5: Qualitative comparison between our method and 2D image-based approaches. The bottom row shows results from the real dataset zheng2024gps. Zoom in for the best view.