GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation

TL;DR

GeneFace++ addresses the challenge of real-time, generalized audio-driven 3D talking face generation by introducing three core ideas: a pitch-aware audio-to-motion pathway to improve lip-sync and temporal coherence, Landmark Locally Linear Embedding to robustly map predicted landmarks into the NeRF renderer’s domain, and an efficient grid-based motion-to-video renderer for fast inference. The approach yields improved objective metrics (LMD, Sync, PSNR, FID) and favorable subjective assessments, achieving near real-time performance (~23.6 FPS) and strong generalization to out-of-domain audio. Extensive ablations demonstrate the necessity and impact of each component, including pitch cues, LLE post-processing, and the hyper-parameter choices in the NeRF conditioning. Overall, GeneFace++ advances NeRF-based talking face generation toward practical, robust applications in real-time digital humans and metaverse contexts.

Abstract

Generating talking person portraits with arbitrary speech audio is a crucial problem in the field of digital human and metaverse. A modern talking face generation method is expected to achieve the goals of generalized audio-lip synchronization, good video quality, and high system efficiency. Recently, neural radiance field (NeRF) has become a popular rendering technique in this field since it could achieve high-fidelity and 3D-consistent talking face generation with a few-minute-long training video. However, there still exist several challenges for NeRF-based methods: 1) as for the lip synchronization, it is hard to generate a long facial motion sequence of high temporal consistency and audio-lip accuracy; 2) as for the video quality, due to the limited data used to train the renderer, it is vulnerable to out-of-domain input condition and produce bad rendering results occasionally; 3) as for the system efficiency, the slow training and inference speed of the vanilla NeRF severely obstruct its usage in real-world applications. In this paper, we propose GeneFace++ to handle these challenges by 1) utilizing the pitch contour as an auxiliary feature and introducing a temporal loss in the facial motion prediction process; 2) proposing a landmark locally linear embedding method to regulate the outliers in the predicted motion sequence to avoid robustness issues; 3) designing a computationally efficient NeRF-based motion-to-video renderer to achieves fast training and real-time inference. With these settings, GeneFace++ becomes the first NeRF-based method that achieves stable and real-time talking face generation with generalized audio-lip synchronization. Extensive experiments show that our method outperforms state-of-the-art baselines in terms of subjective and objective evaluation. Video samples are available at https://genefaceplusplus.github.io .

Figures (7)

• Figure 1: The inference process of GeneFace++. In subfigure (a), we show the overall three-stage pipeline. In subfigure (b), "DA Postnet" denotes the Domain Adaptative Postnet proposed in GeneFace. In subfigure (c), "KNN" denotes finding the K-nearest neighbors of the input landmark, and "Landmark LLE Proj." denotes Landmark Locally Linear Embedding Projection method proposed in Section \ref{['sec:lle']}. In subfigure (d), as for the indexing operation, we perform bi-linear interpolation to query the continuous coordinates in the discrete (spatial/hyper) grids.
• Figure 2: The training process of the Pitch-Aware Audio-to-Motion module. Learnable models are marked with red dotted rectangles and parameter-frozen models are colored in gray. In subfigure (a), Enc. and Dec. denote Encoder and Decoder in VAE, respectively. Disc. means Discriminator. In subfigure (b), the thunder-like symbol represents the "stop gradient" operator. In subfigure(c), "Log-Discretize" denotes the operation that quantizes the log-scale continuous pitch value into discrete tokens.
• Figure 3: The comparison of generated key frame results. We show the speaking word and time step in the demo video. We mark the un-sync and bad rendering quality results with the red and brown arrows, respectively. Please zoom in for better visualization.
• Figure 4: The detailed structure of VAE in the Pitch-Aware Audio-to-Motion module. In subfigure (a), (b), and (c), we show the structure of encoder, prior flow, and the decoder, respectively. Dotted arrows are only operated at the training phase. "Pred." is a shorthand of "predicted".
• Figure 5: The detailed structure of Postnet and its discriminator in the Pitch-Aware Audio-to-Motion module. In subfigure (a), Prior Flow and Decoder are pretrained and fixed to predict raw landmark sequence. We separated them with a blue dotted line. In subfigure (b) and (c) we show the structure of Postnet and its discriminator, respectively. Dotted arrows are only operated at the training phase. "Pred." is a shorthand of "predicted". .