3DPR: Single Image 3D Portrait Relight using Generative Priors

TL;DR

3DPR tackles the ill-posed task of monocular 3D portrait relighting by marrying a 3D-aware generative prior with a novel OLAT-based reflectance model trained on FaceOLAT. It inverts a single input image into a 3D latent space, then synthesizes high-fidelity OLAT images via a triplane reflectance network, which are combined under an HDRI to render relit views from new viewpoints. The FaceOLAT dataset enables learning rich, high-frequency facial reflectance, including hair, and supports multi-view, high-resolution OLAT captures. Quantitatively and qualitatively, 3DPR outperforms state-of-the-art baselines in 3D-consistent view synthesis and physically accurate lighting, with robust performance under sparse and colored illumination and across unseen identities; the authors also release FaceOLAT and code to facilitate further research.

Abstract

Rendering novel, relit views of a human head, given a monocular portrait image as input, is an inherently underconstrained problem. The traditional graphics solution is to explicitly decompose the input image into geometry, material and lighting via differentiable rendering; but this is constrained by the multiple assumptions and approximations of the underlying models and parameterizations of these scene components. We propose 3DPR, an image-based relighting model that leverages generative priors learnt from multi-view One-Light-at-A-Time (OLAT) images captured in a light stage. We introduce a new diverse and large-scale multi-view 4K OLAT dataset of 139 subjects to learn a high-quality prior over the distribution of high-frequency face reflectance. We leverage the latent space of a pre-trained generative head model that provides a rich prior over face geometry learnt from in-the-wild image datasets. The input portrait is first embedded in the latent manifold of such a model through an encoder-based inversion process. Then a novel triplane-based reflectance network trained on our lightstage data is used to synthesize high-fidelity OLAT images to enable image-based relighting. Our reflectance network operates in the latent space of the generative head model, crucially enabling a relatively small number of lightstage images to train the reflectance model. Combining the generated OLATs according to a given HDRI environment maps yields physically accurate environmental relighting results. Through quantitative and qualitative evaluations, we demonstrate that 3DPR outperforms previous methods, particularly in preserving identity and in capturing lighting effects such as specularities, self-shadows, and subsurface scattering. Project Page: https://vcai.mpi-inf.mpg.de/projects/3dpr/

Figures (24)

• Figure 1: Overview of the Dataset. The FaceOLAT lightstage dataset comprises 139 subjects captured from 40 camera viewpoints, resulting in 331 OLAT images per subject, illuminated by point light sources. Each OLAT image is captured at 4K resolution. A snapshot of the dataset is shown in the figure. A detailed description along with dataset demographics is provided in the supplemental material.
• Figure 2: Given one portrait image $\mathbf{C}$, 3DPR renders the subject from a novel viewpoint under new lighting. Left: During the training stage, $\mathbf{C}$ is first fed into the 3D-aware encoder $\mathcal{E}$goae to produce tri-planar features $\mathrm{F_g}$, concatenated with a given light direction $l_i$. Then, the concatenated features are fed into the Reflectance Encoder$\mathrm{R_{enc}}$ and Reflectance Decoder$\mathrm{R_{dec}}$ to render a low-resolution OLAT image $\mathrm{o_{rgb}}$ and high-frequency reflectance features $\mathrm{o_{hf}}$. We further combine $\mathrm{o_{hf}}$ with the corresponding appearance features $\mathrm{c_{hf}}$, to be fed into SR Encoder$\mathrm{E_{SR}}$ to obtain fused high frequency features $\mathrm{p_{hf}}$. At the end, the OLAT Super-Resolution network $\mathrm{SR_{o}}$ produces a high-resolution OLAT image $\hat{\mathbf{O}}$. In our architecture, we use pre-trained networks Chan2022 for StyleGAN and $\mathrm{G_{dec}}$ and keep these modules frozen, and only train $\mathrm{R_{enc}}$, $\mathrm{R_{dec}}$, $\mathrm{E_{SR}}$ and $\mathrm{SR_{o}}$ on light stage dataset. Right: During inference, 3DPR takes a single portrait image, view and light direction as input, and synthesize OLATs, which are then linear combined for novel illumination.
• Figure 3: Simultaneous view synthesis and relighting. The top row shows a reference portrait from FaceOLAT, rendered under selected OLAT directions and corresponding environment map–based relighting, all computed from the 331 OLATs. In the following rows, the first column presents the "in-the-wild" input. Columns 2--4 show OLAT renderings from novel viewpoints, with the light source direction illustrated in the inset. Columns 5--7 show relit outputs from 3DPR, under novel viewpoints and HDRI environment maps, as shown in the insets. This visualization demonstrates 3DPR's ability to simultaneously perform relighting and viewpoint editing on "in-the-wild" images, producing sharp specular highlights, self-shadows and subsurface scattering.
• Figure 4: Baseline Comparisons with PhotoApp mallikarjun2021photoapp, VoRF prao2022vorfprao2023vorf, NeRFFaceLightingnerffacelighting and Lite2Relight prao2024lite2relight. We compare these approaches against $3DPR_{w}{}$ trained on $\textit{WeyrichOLAT}{}$Weyrich2006Analysis. We demonstrate that 3DPR is more effective in preserving the identity of the subjects and produces relighting that more closely resembles the ground truth than other approaches.
• Figure 5: Benefits of OLAT-based Relighting We compare 3DPR with Lite2Relight (L2R) and NeRFFaceLighting (NFL) under increasingly sparse and colored lighting. Each row shows relit results from one method. The first column is the input in-the-wild image; the top row shows the target lighting from a reference subject. The next four columns correspond to sparse white lighting, created by progressively blacking out pixels in an all-white HDR environment map. The final four columns correspond to an indoor environment map, which are made increasingly sparse in a similar manner. As lighting becomes sparser, both baselines degrade noticeably. Generally, NFL suffers from shading artifacts and color inconsistencies due to its low-frequency SH representation and inaccurate lighting-albedo disentanglement. This is more pronounced under colored lights. L2R also struggles to handle sparse conditions, failing to reproduce sharp shadows or specular highlights and thus yielding inconsistent relighting. In contrast, 3DPR maintains identity and reproduces shadows and specular highlights consistently across all scenarios, thanks to its explicit OLAT-based reflectance modeling. These results highlight the robustness and generalization ability of our approach.