Relightable Full-Body Gaussian Codec Avatars

TL;DR

This work tackles relighting of drivable full-body avatars with face and hands under unseen illumination and poses. It introduces Relightable Full-Body Gaussian Codec Avatars, combining orientation-aware diffuse radiance transfer via learnable zonal harmonics, a shadow network to handle non-local occlusions, and deferred shading to achieve high-fidelity specular highlights, all built atop 3D Gaussian Splatting. By predicting per-Gaussian appearance parameters in a local UV-informed frame and rotating them to world space, the method supports efficient, 3D-consistent relighting of articulated bodies, and the irradiance-conditioned shadow network enables realistic non-local shadows. Evaluations on a multi-camera light stage show significant improvements over PBR baselines and ablations, with better generalization to novel illumination and poses and the ability to render eye glints and other high-frequency highlights.

Abstract

We propose Relightable Full-Body Gaussian Codec Avatars, a new approach for modeling relightable full-body avatars with fine-grained details including face and hands. The unique challenge for relighting full-body avatars lies in the large deformations caused by body articulation and the resulting impact on appearance caused by light transport. Changes in body pose can dramatically change the orientation of body surfaces with respect to lights, resulting in both local appearance changes due to changes in local light transport functions, as well as non-local changes due to occlusion between body parts. To address this, we decompose the light transport into local and non-local effects. Local appearance changes are modeled using learnable zonal harmonics for diffuse radiance transfer. Unlike spherical harmonics, zonal harmonics are highly efficient to rotate under articulation. This allows us to learn diffuse radiance transfer in a local coordinate frame, which disentangles the local radiance transfer from the articulation of the body. To account for non-local appearance changes, we introduce a shadow network that predicts shadows given precomputed incoming irradiance on a base mesh. This facilitates the learning of non-local shadowing between the body parts. Finally, we use a deferred shading approach to model specular radiance transfer and better capture reflections and highlights such as eye glints. We demonstrate that our approach successfully models both the local and non-local light transport required for relightable full-body avatars, with a superior generalization ability under novel illumination conditions and unseen poses.

Figures (8)

• Figure 1: Overview of our approach. Given a body latent code $\mathbf{l}_b$ and a face latent code $\mathbf{l}_f$ computed by a keypoint encoder and canonicalized viewing directions $\hat{\mathbf{\omega}}_o$ as input, we decode the geometry parameters of 3D Gaussians $\{ \mathbf{R}_k, \mathbf{s}_k, \mathbf{t}_k, o_k \}$ (Sec. \ref{['sec:geometry']}), and appearance parameters consisting of light transport coefficients $\{\mathbf{z}_k^c, \mathbf{z}_k^m\}$, normals $\{ \mathbf{n}_k \}$, roughness $\{\sigma_k \}$, and specular visibility $\{ v_k \}$ (Sec. \ref{['sec:appearance']}). We integrate the light with diffuse light transport coefficients to yield per-Gaussian diffuse color, while using deferred shading to compute specular color. The final color is modulated by a shadow map predicted by a shadow network (Sec. \ref{['sec:shadowing']}).
• Figure 2: Our appearance model vs. PBR appearance model. The PBR appearance model fails to capture subsurface scattering effects for skins and translucent structures such as hairs. It also produces a darker appearance for concave structures such as ears by omitting global illumination.
• Figure 3: ZH vs. SH for diffuse light transport. Note the incorrect shading on the right arm in the SH variant.
• Figure 4: Qualitative results shadow networks. The learned light transport is not sufficient to capture the shadowing effects caused by body articulation without the help of the shadow network.
• Figure 5: Capture Dome. Our multi-camera light stage with 512 cameras and 1024 controllable light sources. The dome has a radius of $2.75$ meters. Each camera has 24 mega-pixels resolution and records video with up to 90Hz.