Universal Facial Encoding of Codec Avatars from VR Headsets

TL;DR

The paper tackles the challenge of real-time, photorealistic avatar animation for VR telepresence across diverse users and environments, where facial observations are incomplete and headset-induced distortions are prevalent. It introduces a universal facial encoding framework built upon a universal prior model, augmented decoders with explicit eyeball modeling, and a calibration-driven encoder that uses anchor expressions, all trained with a novel self-supervised pretraining regime on large-scale unlabeled head-mounted camera data via novel-view MAE reconstruction. Key contributions include ground-truth supervision for HMC inputs under varying illumination, a calibration-conditioned encoding architecture with anchor expressions that improves generalization, and a self-supervised pretraining pipeline that substantially reduces distortion relative to prior methods (demonstrated by a $>20\%$ improvement in quantitative metrics). The approach enables high-fidelity, low-latency live VR calls, robust to donning and lighting variations, and capable of retargeting to unseen avatars; however, challenges remain in subtler lip motions, tongue tracking, and potential latent-code out-of-distribution issues. Overall, the work advances universal, real-time codec avatars for VR telepresence with practical impact for immersive communication and new avenues for multimodal sensing.

Abstract

Faithful real-time facial animation is essential for avatar-mediated telepresence in Virtual Reality (VR). To emulate authentic communication, avatar animation needs to be efficient and accurate: able to capture both extreme and subtle expressions within a few milliseconds to sustain the rhythm of natural conversations. The oblique and incomplete views of the face, variability in the donning of headsets, and illumination variation due to the environment are some of the unique challenges in generalization to unseen faces. In this paper, we present a method that can animate a photorealistic avatar in realtime from head-mounted cameras (HMCs) on a consumer VR headset. We present a self-supervised learning approach, based on a cross-view reconstruction objective, that enables generalization to unseen users. We present a lightweight expression calibration mechanism that increases accuracy with minimal additional cost to run-time efficiency. We present an improved parameterization for precise ground-truth generation that provides robustness to environmental variation. The resulting system produces accurate facial animation for unseen users wearing VR headsets in realtime. We compare our approach to prior face-encoding methods demonstrating significant improvements in both quantitative metrics and qualitative results.

Figures (18)

• Figure 1: Camera placements of the VR headset and example views. (a) Tracking headset: an off-the-shelf VR headset that the universal encoder uses during live calls. The oblique mouth views are challenging for extracting precise facial motion; (b) Training headset: we augment the tracking headset with additional cameras to establish facial encoding that correspond to the training views for training encoders.
• Figure 2: System overview. We augment universal avatar decoders to make it amenable to the style transfer method of finding HMC-avatar correspondences. We then present two methods to boost model generalization: expression calibration and self-supervised learning on unlabelled data.
• Figure 3: Sensing challenges in the HMC images from the off-the-shelf VR headset we use in this work questpro (subfigure (a)-(c)). Compared to those used in prior works (subfigure (d) and (e)), our data suffers from more obliqueness, incompleteness, and extreme illumination.
• Figure 4: Establishing HMC-avatar correspondences. We build upon the analysis-by-synthesis method presented in prior work schwartz2020eyes, and address a severe limitation in handling extreme illumination changes, by regressing a "lighting code" to modulate style transfer in both texture and image spaces, via predicting low-resolution gain/bias maps by deconvolutional networks $\mathcal{L}_\text{im}$ and $\mathcal{L}_\text{tex}$.
• Figure 5: Examples of established HMC-avatar correspondences. In each grid, we show HMC images on a subset of views on the training headset (left), decoder rendering $I_{\text{mesh}}$ using found expression code $\mathbf{e}^\star$ and HMC-relative viewpoint $\mathbf{v}^\star$ with textures $\mathbf{T}$ (center), and frontal rendering $I_\text{vol}$ with volumetric slabs $\mathbf{M}$ (right).