REFA: Real-time Egocentric Facial Animations for Virtual Reality

TL;DR

The paper tackles real-time egocentric facial animation in VR by embedding five infrared cameras in a headset to capture facial motion, obviating manual calibration. It introduces a holistic pipeline that learns to predict 3D blendshape coefficients using a subject-specific rig derived from lightweight captures, aided by a differentiable renderer to generate pseudo ground truth and augmented by synthetic data and artist priors. A novel iterative distillation framework and domain-adaptive training align real and synthetic data, yielding an on-device model with auto-calibration and failure detection. The approach advances VR social presence by delivering accurate, non-intrusive facial tracking suitable for conferencing, gaming, and remote collaboration, with potential future enhancements leveraging audio and temporal cues. The blendshape representation comprises $53$ bases with coefficient vectors $\boldsymbol{b} \in \mathbb{R}^{53}$, eye gaze $\mathbf{g}_\mathrm{l},\mathbf{g}_\mathrm{r} \in \mathbb{R}^2$, and rigid motion $(\mathbf{R},\mathbf{t})$, all processed at $400\times400$ resolution and $30$ Hz in real time.

Abstract

We present a novel system for real-time tracking of facial expressions using egocentric views captured from a set of infrared cameras embedded in a virtual reality (VR) headset. Our technology facilitates any user to accurately drive the facial expressions of virtual characters in a non-intrusive manner and without the need of a lengthy calibration step. At the core of our system is a distillation based approach to train a machine learning model on heterogeneous data and labels coming form multiple sources, \eg synthetic and real images. As part of our dataset, we collected 18k diverse subjects using a lightweight capture setup consisting of a mobile phone and a custom VR headset with extra cameras. To process this data, we developed a robust differentiable rendering pipeline enabling us to automatically extract facial expression labels. Our system opens up new avenues for communication and expression in virtual environments, with applications in video conferencing, gaming, entertainment, and remote collaboration.

Figures (10)

• Figure 1: Our HMD is equipped with five face cameras, two for eye and eyebrow regions, two for mouth, and one for glabella. Note that we mirror the left eye and left mouth images. A multitude of camera configurations have been considered during the design of the HMD. Among these configurations, the one highlighted in green has been implemented, which has better visibility and range of motion metrics than the configurations highlighted in orange or red (Orange or red configurations seems to have better visibility in glabella, but they pose conflicts with users' glasses frames and the HMD's Inter-pupil distance adjustment mechanism).
• Figure 2: Our data collection HMD is equipped with additional five cameras, offering better visibility of the face than the embedded cameras. This camera setup allows us to improve the quality of the generated pseudo ground truth.
• Figure 3: Our expression fitting pipeline takes RGBD frames as input and proceeds through a series of steps to generate a fitted mesh as output. We use this process to fit a set of facial expressions, from which we then create a subject-specific rig using example-based facial rigging li2010example.
• Figure 4: System diagram of estimating blendshape coefficients from HMD images, based on the subject-specific blendshape rig.
• Figure 5: Example of a synthetic frame used to train our ML model