NeRFFaceSpeech: One-shot Audio-driven 3D Talking Head Synthesis via Generative Prior

TL;DR

NeRFFaceSpeech tackles the data-intensive bottleneck of NeRF-based audio-driven talking heads by enabling 3D-consistent synthesis from a single image. It combines generative priors (StyleNeRF) with a 3DMM-driven feature space and introduces ray deformation for audio-conditioned dynamics, complemented by LipaintNet to recover inner-mouth details in a self-supervised manner. The method includes a spatial synchronization step, a blending pipeline, and a comprehensive evaluation showing robustness to pose changes and superior perceptual quality, as demonstrated by user studies. This approach enables scalable, 3D-aware talking head synthesis suitable for VR/AR and other immersive applications without requiring extensive identity-specific video data.

Abstract

Audio-driven talking head generation is advancing from 2D to 3D content. Notably, Neural Radiance Field (NeRF) is in the spotlight as a means to synthesize high-quality 3D talking head outputs. Unfortunately, this NeRF-based approach typically requires a large number of paired audio-visual data for each identity, thereby limiting the scalability of the method. Although there have been attempts to generate audio-driven 3D talking head animations with a single image, the results are often unsatisfactory due to insufficient information on obscured regions in the image. In this paper, we mainly focus on addressing the overlooked aspect of 3D consistency in the one-shot, audio-driven domain, where facial animations are synthesized primarily in front-facing perspectives. We propose a novel method, NeRFFaceSpeech, which enables to produce high-quality 3D-aware talking head. Using prior knowledge of generative models combined with NeRF, our method can craft a 3D-consistent facial feature space corresponding to a single image. Our spatial synchronization method employs audio-correlated vertex dynamics of a parametric face model to transform static image features into dynamic visuals through ray deformation, ensuring realistic 3D facial motion. Moreover, we introduce LipaintNet that can replenish the lacking information in the inner-mouth area, which can not be obtained from a given single image. The network is trained in a self-supervised manner by utilizing the generative capabilities without additional data. The comprehensive experiments demonstrate the superiority of our method in generating audio-driven talking heads from a single image with enhanced 3D consistency compared to previous approaches. In addition, we introduce a quantitative way of measuring the robustness of a model against pose changes for the first time, which has been possible only qualitatively.

Figures (4)

• Figure 1: Overall pipeline of our method: Given a single image and an audio input, the preprocessing stage extracts 3DMM parameters and 3D features in the NeRF space. Subsequent spatial synchronization between 3DMM and feature space enables the reflection of expression vertex changes within the feature space. These changes are computed in accordance with the audio time step $t$, resulting in the generation of facial movements. The final output frame emerges from feature blending, where the deformed features are merged with the inner-mouth details generated by the proposed LipaintNet.
• Figure 2: Visual comparison with outputs of baselines.
• Figure 3: Visual comparison of the robustness to pose changes.
• Figure 4: Ablation study for proposed LipaintNet and Average Mask methods.