Fast and Physically-based Neural Explicit Surface for Relightable Human Avatars

TL;DR

PhyNES addresses relightable human avatars from sparse-view videos by marrying Neural Explicit Surface with pose-conditioned 2D neural maps for geometry, texture, and material. It introduces a rasterization-based neural renderer and dedicated material networks (albedo and roughness) to enable efficient physically-based rendering and relighting, guided by learnable light probes and a Cook-Torrance BRDF. Two training stages optimize pose-dependent geometry/texture and material properties, while a mesh loss regularizes geometry and a hash-encoded MLP accelerates training. Experiments on real and synthetic datasets show competitive relighting quality with substantially faster rendering and lower memory usage than vector/volume-based approaches, as well as improved mesh reconstruction. The approach offers practical benefits for real-time AR/VR and streaming applications by providing a compact, query-efficient representation that supports flexible illumination without heavy computational costs.

Abstract

Efficiently modeling relightable human avatars from sparse-view videos is crucial for AR/VR applications. Current methods use neural implicit representations to capture dynamic geometry and reflectance, which incur high costs due to the need for dense sampling in volume rendering. To overcome these challenges, we introduce Physically-based Neural Explicit Surface (PhyNES), which employs compact neural material maps based on the Neural Explicit Surface (NES) representation. PhyNES organizes human models in a compact 2D space, enhancing material disentanglement efficiency. By connecting Signed Distance Fields to explicit surfaces, PhyNES enables efficient geometry inference around a parameterized human shape model. This approach models dynamic geometry, texture, and material maps as 2D neural representations, enabling efficient rasterization. PhyNES effectively captures physical surface attributes under varying illumination, enabling real-time physically-based rendering. Experiments show that PhyNES achieves relighting quality comparable to SOTA methods while significantly improving rendering speed, memory efficiency, and reconstruction quality.

Figures (4)

• Figure 1: The overall rendering and relighting pipeline of PhyNES includes two learning phases, as shown in (a) and (b). (a) In the initial stage, the fitted SMPL model will be used as a reference plane to transform volume sampling points' world coordinates to a thin layer of transformed $uv$ coordinate space. $uv$ and the SMPL pose parameters $\theta$ are used as input to a hash surface offset network and a dynamic texture network. These estimators predict the offset $l$ and color $c$ at the corresponding UV coordinates, and the signed distance is computed using a conversion module. A pose-dependent dynamic mesh (both in surface offset and textures) will be generated. (b) An optimizable light probe array will be configured around the stage in the second phase. The surface normals can be conveniently computed with the rasterization-based neural renderer for each learnable probe, incidence, and observation direction. An albedo and roughness network will predict respective surface material attributes for each rasterized coordinate $uv$ to facilitate relighting applications. Our model produces the final physically-based rendering output by connecting these attributes to a differentiable BRDF cook1982reflectance function.
• Figure 2: Qualitative comparison in (a) seen-pose rendering and (b) novel-pose rendering tasks on the ZJU-MoCap dataset. As indicated in red squares, PhyNES can better model dynamic variations.
• Figure 3: Comparative visualization of surface normals based on the reconstructed geometry by different methods on the SyntheticHuman dataset.
• Figure 4: Qualitative Comparisons with RA and Relighting4D on albedo, visibility, and rendering under four novel lights on two different synthetic humans.