LegacyAvatars: Volumetric Face Avatars For Traditional Graphics Pipelines

TL;DR

LegacyAvatars presents a ML-free pipeline that exports volumetric face avatars as a static layered mesh with UV-space warp and texture bases, enabling efficient, shader-based rendering on legacy graphics pipelines. By discretizing geometry, appearance, and deformation into $N$ implicit surfaces and linear blends of per-frame expression coefficients, it achieves real-time rendering with simple rasterization and streaming suitable for WebGL and consumer devices. The approach demonstrates competitive quality against modern volumetric methods while offering native compatibility, ease of streaming, and broad deployability, with quantified results showing strong PSNR/SSIM/LPIPS performance and favorable Web metrics. This work significantly lowers the barrier to practical telepresence and avatar deployment across platforms by aligning advanced volumetric rendering with traditional graphics infrastructure.

Abstract

We introduce a novel representation for efficient classical rendering of photorealistic 3D face avatars. Leveraging recent advances in radiance fields anchored to parametric face models, our approach achieves controllable volumetric rendering of complex facial features, including hair, skin, and eyes. At enrollment time, we learn a set of radiance manifolds in 3D space to extract an explicit layered mesh, along with appearance and warp textures. During deployment, this allows us to control and animate the face through simple linear blending and alpha compositing of textures over a static mesh. This explicit representation also enables the generated avatar to be efficiently streamed online and then rendered using classical mesh and shader-based rendering on legacy graphics platforms, eliminating the need for any custom engineering or integration.

Figures (14)

• Figure 1: We present a novel representation for rendering animatable volumetric 3D face avatars using meshes and textures. From an enrollment sequence of a subject, we learn a layered mesh and blend-textures that model the geometry, appearance and deformations, and a simple linear transformation that maps tracked face model parameters to blend weights. Our representation is readily compatible with existing streaming infrastructure and can be deployed in traditional graphics pipelines in a device- and platform-agnostic way.
• Figure 2: Training pipeline for enrollment phase. Our model consists of three separate modules: a manifold predictor $\mathcal{M}$, a warp predictor $\mathcal{W}$, and a texture predictor $\mathcal{T}$. Here, $\mathcal{M}$ is a scalar field that defines layered implicit surfaces. The intersections with these surfaces are spherically mapped to the UV-space via a learnable function $f$. Then, the output subsequently queries $\mathcal{W}$ to obtain a basis of UV-offsets. These offsets are then linearly blended as a function of expression parameters of a face model and added to the original values. Finally, the new coordinates are fed through $\mathcal{T}$, which predicts a basis of RGBA appearances that are also linearly blended as a function of expression parameters. Each module takes in learned latent codes $\phi_m, \phi_w, \phi_t$ for multi-subject training, while $\mathcal{W}$ and $\mathcal{T}$ take in learnable embedding matrices $E_w$ and $E_t$ to output bases of warps and textures.
• Figure 3: Radiance decomposition. During training, the radiance is modeled using spherical harmonics coefficients, which can be decomposed into diffuse (view-independent) and specular (view-dependent) components. The appearance can be exported as just diffuse or both diffuse and specular texture images.
• Figure 4: UV-space predictions. Given a ground truth per-pixel UV-coordinates $\mathbf{u}_\mathrm{gt}$, our model is supervised to match the expectation of warped coordinates $\bar{\mathbf{u}}'$ to the ground truth. We visualize the expectations of the spherically mapped coordinates $\bar{\mathbf{u}}$ and the warps $\delta\bar{\mathbf{u}}$ for reference.
• Figure 5: Programmable shader. At the deployment phase, our 3D assets (a single static layered mesh and bases of warp and texture maps) can easily be used to render dynamic and volumetric faces via a programmable shader on any graphics platform.