Polarimetric BSSRDF Acquisition of Dynamic Faces

TL;DR

This work addresses the challenge of capturing and rendering dynamic, translucent human faces by introducing a polarimetric BSSRDF that explicitly models heterogeneous subsurface scattering and biophysical skin parameters. It combines multispectral polarimetric imaging with a two-layer skin model to recover per-texel refractive index, specular and single-scattering properties, and diffusion-driven subsurface effects, enabling space-time coherent geometry and appearance maps. A two-stage reconstruction pipeline is proposed: a static initialization over multiple views to estimate detailed geometry and polarimetric parameters, followed by per-frame optimization to recover bio-physiological maps (e.g., melanin and hemoglobin) and dynamic skin appearance, all within a differentiable inverse rendering framework. Validation across 11 subjects demonstrates accurate spectral reflectance, refractive indices, and polarimetric reflectance, with qualitative and quantitative improvements over prior static and non-polarimetric face capture methods, and enables realistic appearance editing and rendering integration.

Abstract

Acquisition and modeling of polarized light reflection and scattering help reveal the shape, structure, and physical characteristics of an object, which is increasingly important in computer graphics. However, current polarimetric acquisition systems are limited to static and opaque objects. Human faces, on the other hand, present a particularly difficult challenge, given their complex structure and reflectance properties, the strong presence of spatially-varying subsurface scattering, and their dynamic nature. We present a new polarimetric acquisition method for dynamic human faces, which focuses on capturing spatially varying appearance and precise geometry, across a wide spectrum of skin tones and facial expressions. It includes both single and heterogeneous subsurface scattering, index of refraction, and specular roughness and intensity, among other parameters, while revealing biophysically-based components such as inner- and outer-layer hemoglobin, eumelanin and pheomelanin. Our method leverages such components' unique multispectral absorption profiles to quantify their concentrations, which in turn inform our model about the complex interactions occurring within the skin layers. To our knowledge, our work is the first to simultaneously acquire polarimetric and spectral reflectance information alongside biophysically-based skin parameters and geometry of dynamic human faces. Moreover, our polarimetric skin model integrates seamlessly into various rendering pipelines.

Figures (16)

• Figure 1: Spectral absorption coefficients of each biophysical component in the human skin: oxy-hemoglobin, deoxy-hemoglobin, eumelanin, pheomelanin, and skin base.
• Figure 2: Our multispectral polarimetric imaging setup. (a) Our system consists of a multispectral polarization module (yellow box) that captures four different linear polarization angles with six multispectral channels and four 3D stereo imaging modules (green boxes). (b) A closeup image of the stereo module composed of two machine vision cameras. Stereo pairs are used to acquire dense depth maps to obtain a complete geometry model and its corresponding texture mapping for each frame. (c)--(e) Closeup images of the multispectral polarization camera module equipped with two polarization cameras (d) covered with two different Dolby filters and 40 LED lights, separated into eight modules of five LEDs, each shown in (e). Our multispectral polarization module captures six different multispectral channels using two different Dolby filters for three RGB channels. Each light is covered with a vertical polarization filter. (f) Schematic diagram of the system configuration. The module also captures four different orientations of linear polarization. The eight light modules are placed at a distance of approximately 10 cm. As the subject is placed 100 cm away, light and camera become almost coaxial at a 5.72$^\circ$ angle. These cameras are synchronized through GPIO cables. We capture images at 20 fps.
• Figure 3: Multispectral-channel calibration and light spectral power distribution (black line). We disassemble left and right glass lenses from Dolby multispectral 3D anaglyph glasses to use each as a bandpass filter in front of two sRGB cameras. Each color channel is subdivided into two sub-color channels, resulting in six multispectral channels.
• Figure 4: Overview of our reconstruction algorithm. It consists of two stages: first, the initialization takes multiple views of a static face (top). We then perform a per-frame optimization from a single view (bottom). Both stages follow a similar procedure: first, we obtain a mesh from stereo pairs (left), then we iteratively optimize a displacement map and a set of polarimetric appearance parameters (middle), and lastly, we use inverse rendering to iteratively optimize the biophysical parameters of human skin (right).
• Figure 5: Our dynamic face reconstruction results. We record data from 11 individuals, each exhibiting diverse skin tones, gender identities, and ethnic backgrounds. Our method successfully captures polarimetric reflectance parameters, biophysical parameters, and geometry with high accuracy. Refer to the supplemental document (Appendix A) and video for more results of dynamic faces.