RelightAnyone: A Generalized Relightable 3D Gaussian Head Model

TL;DR

RelightAnyone tackles the problem of relighting head avatars reconstructed with 3D Gaussian Splatting by separating identity/lighting from relighting. It introduces a two-stage pipeline: Stage1 learns a multi-identity full-on Gaussian avatar from flat-lit multi-view data using an identity code and a dataset-specific lighting code, producing a dense Gaussian field over a 1024×1024 UV texture map. Stage2 trains a UNet to map these full-on Gaussian colors to relightable RGCA parameters, enabling physically-based relighting under arbitrary illumination with reduced OLAT data requirements. The Stage1 prior generalizes across diverse flat-lit datasets, while Stage2 leverages a smaller OLAT corpus and self-supervised lighting alignment to transfer relighting capabilities to new subjects, including those seen in-the-wild from a single image. Across qualitative and quantitative evaluations, RelightAnyone outperforms contemporary 3D GAN-based and 2D diffusion-based relighting methods in view-consistent relighting and identity preservation, though it still struggles with hair, accessories, and expression dynamics that future work could address.

Abstract

3D Gaussian Splatting (3DGS) has become a standard approach to reconstruct and render photorealistic 3D head avatars. A major challenge is to relight the avatars to match any scene illumination. For high quality relighting, existing methods require subjects to be captured under complex time-multiplexed illumination, such as one-light-at-a-time (OLAT). We propose a new generalized relightable 3D Gaussian head model that can relight any subject observed in a single- or multi-view images without requiring OLAT data for that subject. Our core idea is to learn a mapping from flat-lit 3DGS avatars to corresponding relightable Gaussian parameters for that avatar. Our model consists of two stages: a first stage that models flat-lit 3DGS avatars without OLAT lighting, and a second stage that learns the mapping to physically-based reflectance parameters for high-quality relighting. This two-stage design allows us to train the first stage across diverse existing multi-view datasets without OLAT lighting ensuring cross-subject generalization, where we learn a dataset-specific lighting code for self-supervised lighting alignment. Subsequently, the second stage can be trained on a significantly smaller dataset of subjects captured under OLAT illumination. Together, this allows our method to generalize well and relight any subject from the first stage as if we had captured them under OLAT lighting. Furthermore, we can fit our model to unseen subjects from as little as a single image, allowing several applications in novel view synthesis and relighting for digital avatars.

Figures (17)

• Figure 1: We present RelightAnyone, a method for reconstructing and relighting head avatars from multi-view (top row) or single images in-the-wild (bottom row), trained on a limited dataset of multi-view OLAT faces and larger datasets of mulit-view flat-lit faces.
• Figure 2: Pipeline of RelightAnyone: We first train a Stage 1 network that, given a learned subject identity code and a learned lighting code that distinguishes between different illuminations of different datasets, predicts a Gaussian avatar under full-on lighting, where the parameters of the Gaussian primitives are encoded in UV texture maps. Then, the Stage 2 relighting network maps the full-on Gaussian parameter textures from Stage 1 to RGCA parameters to allow for relighting of the avatar.
• Figure 3: Reconstruction and relighting examples from multi-view (rows 1-5) and single images in the wild (rows 6-7). From top to bottom, the first two subjects are from D2; subject 3 is from D3; subject 4 and 5 are from D4, and subject 6 and 7 are self-captured in-the-wild images. Our method can accurately reconstruct a detailed and multi-view stable Gaussian avatar in Stage 1 (column "Reconstruction") that can be relit with arbitrary environment maps using Stage 2, even under harsh outdoor illumination. Images best viewed zoomed-in.
• Figure 4: For a single image of an unseen subject in-the-wild (a), our method predicts a Gaussian avatar (b) with an intrinsic decomposition into diffuse albedo (c), diffuse shading (d), specular component (e), and specular normals (f).
• Figure 5: Ablation of the two-stage pipeline on the test subjects in D1 under OLAT lighting. The single-stage pipeline especially struggles in the shadowed areas. The render errors are in the range of $-0.1$0.1. Images best viewed zoomed-in.