Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image

TL;DR

Holo-Relighting tackles the problem of controllable relighting and free-view portrait synthesis from a single image. It leverages a pretrained 3D GAN and performs inversion to obtain geometry via the tri-plane generator $G_{tri}$ with latent code $w^{*}$, then uses a relighting network conditioned on a target environment map $E$, head pose, and camera pose to produce a shading tri-plane $f^{s}_{triplane}$ that is added to the albedo tri-plane $f^{a}_{triplane}$ to form the relit tri-plane $f^{r}_{triplane}$ for volume rendering. Two data-rendering strategies—Multi-View Regularization and Portrait Shading Transfer—improve geometry fidelity and shading alignment during training on OLAT data. The method delivers state-of-the-art relighting quality with strong 3D consistency and controllability, capable of non-Lambertian effects such as specular highlights and cast shadows, and enables applications like shadow diffusion from a single portrait. Overall, it enables practical, flexible portrait editing with few capture constraints.

Abstract

At the core of portrait photography is the search for ideal lighting and viewpoint. The process often requires advanced knowledge in photography and an elaborate studio setup. In this work, we propose Holo-Relighting, a volumetric relighting method that is capable of synthesizing novel viewpoints, and novel lighting from a single image. Holo-Relighting leverages the pretrained 3D GAN (EG3D) to reconstruct geometry and appearance from an input portrait as a set of 3D-aware features. We design a relighting module conditioned on a given lighting to process these features, and predict a relit 3D representation in the form of a tri-plane, which can render to an arbitrary viewpoint through volume rendering. Besides viewpoint and lighting control, Holo-Relighting also takes the head pose as a condition to enable head-pose-dependent lighting effects. With these novel designs, Holo-Relighting can generate complex non-Lambertian lighting effects (e.g., specular highlights and cast shadows) without using any explicit physical lighting priors. We train Holo-Relighting with data captured with a light stage, and propose two data-rendering techniques to improve the data quality for training the volumetric relighting system. Through quantitative and qualitative experiments, we demonstrate Holo-Relighting can achieve state-of-the-arts relighting quality with better photorealism, 3D consistency and controllability.

Figures (9)

• Figure 1: Holo-Relighting performs volumetric relighting on a single input portrait image, allowing users to individually control (1) lighting effect via an environment map, (2) camera viewpoint and (3) head pose. It is highly expressive and can render complex illumination effects on in-the-wild human faces with accurate view consistency (a ). Controls are well disentangled to produce a realistic rendering of moving shadows cast by a point light while rotating the head (b). Practical photographic applications such as shadow diffusion (softening) are also made feasible with our system (c).
• Figure 2: An overview of Holo-Relighting. Our method consists of three stages. (a) We first remove the shading from the input portrait and estimate an albedo image. (b) We then conduct GAN inversion upon EG3D to obtain a latent code $w^{*}$ encoding 3D information of the subject. (c) The relighting network takes in the lighting condition, head pose as well as intermediate GAN features produced by EG3D's tri-plane generator $G_{tri}$ using the inverted latent code $w^{*}$, and predicts a shading tri-plane $f^{s}_{triplane}$, which is summed to the albedo tri-plane $f^{a}_{triplane}$, resulting in the relit tri-plane $f^{r}_{triplane}$ with lighting embedded. High-resolution RGB images can be rendered from $f^{r}_{triplane}$ via volume rendering and a super-resolution network. During training, we freeze $G_{tri}$ and only update the relighting net.
• Figure 3: Illustration of proposed portrait shading transfer. We create a pseudo ground-truth image $\Tilde{\mathcal{R}}_{gt}$ by transferring the shading from the OLAT renders. $\Tilde{\mathcal{R}}_{gt}$ contains target illumination, and is consistent with the appearance encoded in $f_{GAN}$ (i.e.$\mathcal{A}_{inv}$).
• Figure 4: Visual comparisons of free-view relighting on in-the-wild portraits. Environment maps are shown as insets. We provide a reference image (last column) rendered using OLAT subject with the target lighting as guidance for comparison.
• Figure 5: Visual comparisons against free-view relighting methods on test set. Our method produces more faithful target lighting effects.