Audio Driven Real-Time Facial Animation for Social Telepresence

TL;DR

This work tackles real-time photorealistic facial telepresence by converting audio into latent facial expressions that drive universal 3D avatars. It introduces an online Transformer-based diffusion encoder and a distillation-based single-step generator to achieve real-time performance (<15 ms GPU time) while maintaining natural lip sync and expressive detail. The system supports multimodal extensions, including emotion conditioning and VR headset sensor inputs, and demonstrates significant accuracy and latency improvements over offline baselines, with live VR demonstrations. These advances enable accessible, cross-identity social telepresence in virtual environments, while acknowledging limitations and societal considerations for responsible deployment.

Abstract

We present an audio-driven real-time system for animating photorealistic 3D facial avatars with minimal latency, designed for social interactions in virtual reality for anyone. Central to our approach is an encoder model that transforms audio signals into latent facial expression sequences in real time, which are then decoded as photorealistic 3D facial avatars. Leveraging the generative capabilities of diffusion models, we capture the rich spectrum of facial expressions necessary for natural communication while achieving real-time performance (<15ms GPU time). Our novel architecture minimizes latency through two key innovations: an online transformer that eliminates dependency on future inputs and a distillation pipeline that accelerates iterative denoising into a single step. We further address critical design challenges in live scenarios for processing continuous audio signals frame-by-frame while maintaining consistent animation quality. The versatility of our framework extends to multimodal applications, including semantic modalities such as emotion conditions and multimodal sensors with head-mounted eye cameras on VR headsets. Experimental results demonstrate significant improvements in facial animation accuracy over existing offline state-of-the-art baselines, achieving 100 to 1000 times faster inference speed. We validate our approach through live VR demonstrations and across various scenarios such as multilingual speeches.

Figures (13)

• Figure 1: Overview of the decoder (left) and the encoder-decoder pipeline (right). The encoder $\mathcal{E}$ generates expression codes in real time based on single-step denoising of a diffusion model, which is decoded into 3DGS and mesh by the decoder $\mathcal{D}$.
• Figure 2: Transformer-based denoising network architecture. Windowed attention mask is applied in the self-attention layer of each Transformer block.
• Figure 3: Pipeline of distillation training. The distilled model first learns via $\mathcal{L}_\text{distill}$ from the original model's multi step and its own single step outputs, with re-noised samples are used to compute $\mathcal{L}_\text{DMD}$.
• Figure 4: Emotion conditioning architecture (left) and training pipeline (right). Zero convolutional layers are added between the linear and Transformer blocks for conditioning. These layers are trained with $\mathcal{L}_\text{face}$ from emotion-conditioned model and $\mathcal{L}_\text{lip}$ from the neutral model's lip motion.
• Figure 5: Pipeline for multimodal applications using a VR headset, with 2 HMC eye cameras and a microphone. Eye features are extracted from HMC images, and are given as a input with the audio signal.