Gaussian Shadow Casting for Neural Characters

TL;DR

This work addresses the challenge of explicit lighting and hard shadows in neural character reconstruction under directional illumination by introducing Gaussian Shadow Casting (GSC), a differentiable, skeleton-attached anisotropic Gaussian density proxy that enables closed-form shadow integration. Coupled with a deferred neural illumination pipeline, the method separates albedo, shading, and shadows, optimizes light direction without supervision, and supports relighting with environment maps. Empirical results on synthetic and outdoor datasets show improved novel-pose rendering, outdoor shadow realism, and HDRi relighting, with favorable training and runtime efficiency compared to NeRF-based shadow methods. The approach advances practical, controllable neural characters for CG/entertainment, offering robust relighting and scene integration in diverse illumination conditions.

Abstract

Neural character models can now reconstruct detailed geometry and texture from video, but they lack explicit shadows and shading, leading to artifacts when generating novel views and poses or during relighting. It is particularly difficult to include shadows as they are a global effect and the required casting of secondary rays is costly. We propose a new shadow model using a Gaussian density proxy that replaces sampling with a simple analytic formula. It supports dynamic motion and is tailored for shadow computation, thereby avoiding the affine projection approximation and sorting required by the closely related Gaussian splatting. Combined with a deferred neural rendering model, our Gaussian shadows enable Lambertian shading and shadow casting with minimal overhead. We demonstrate improved reconstructions, with better separation of albedo, shading, and shadows in challenging outdoor scenes with direct sun light and hard shadows. Our method is able to optimize the light direction without any input from the user. As a result, novel poses have fewer shadow artifacts and relighting in novel scenes is more realistic compared to the state-of-the-art methods, providing new ways to pose neural characters in novel environments, increasing their applicability.

Figures (13)

• Figure 1: Gaussian Shadow Casting (GSC): Our method is able to reconstruct 3D neural characters from a sparse set of videos in settings with strong directional illumination. GSC uses a sum of Gaussians density model to cast secondary shadow rays efficiently with an analytic formula. Our method learns to remove shadows from the neural color field, allowing us to relight in novel illuminations. All faces are blurred for anonymity.
• Figure 2: Method Overview. Our method takes as input images and poses of a person. Using a neural radiance field as a backbone$^{1}$su2022danbo, density, normals, and albedo values are volumetrically reconstructed and rendered. We fit a sum of 3D anisotropic Gaussian density model to approximate the neural density field and compute shadow maps using our novel anisotropic Gaussian ray occlusion equations. The shadow map is combined with a diffuse shading pass to produce the lit image. The whole model is optimized with a photometric loss against the training images. Our method is able to optimize the light direction and ambient intensity without any initialization. It also separates albedo from shading and shadow, allowing us to relight the model.
• Figure 3: Gaussian Density Model. The approximation to the NeRF's density field using a sum of 3D anisotropic Gaussians using: a) 2 Gaussians per bone, b) 4 Gaussian per bone, and c) 8 Gaussian per bone; d) is the groundtruth mesh. Note: ellipses are scaled to 2.5 STD of the Gaussians (99$^{\text{th}}$ percentile)
• Figure 4: 3D Anisotropic Gaussian Raytracing. a) A cross-section of a 3D anisotropic Gaussian with rays passing through the Gaussian. b) The computed 1D Gaussians resulting from our derivation in Section \ref{['sec:gaussianray']} (colored solid), compared to sampling the 3D Gaussian directly (dashed), with their exact match validating the correctness. c) The transmittance along each ray which is used as the shadow map value.
• Figure 5: Qualitative Comparison (train). Our method is able to better estimate albedo where shadows are not baked in as part of the neural field. We compare against Relighting4D on training images as their underlying neural body model is unable to handle novel poses.