NeRF-AD: Neural Radiance Field with Attention-based Disentanglement for Talking Face Synthesis

TL;DR

An Attention-based Disentanglement module is introduced to disentangle the face into Audio-face and Identity-face using speech-related facial action unit (AU) information, and the NeRF-AD outperforms state-of-the-art methods in generating realistic talking face videos, including image quality and lip synchronization.

Abstract

Talking face synthesis driven by audio is one of the current research hotspots in the fields of multidimensional signal processing and multimedia. Neural Radiance Field (NeRF) has recently been brought to this research field in order to enhance the realism and 3D effect of the generated faces. However, most existing NeRF-based methods either burden NeRF with complex learning tasks while lacking methods for supervised multimodal feature fusion, or cannot precisely map audio to the facial region related to speech movements. These reasons ultimately result in existing methods generating inaccurate lip shapes. This paper moves a portion of NeRF learning tasks ahead and proposes a talking face synthesis method via NeRF with attention-based disentanglement (NeRF-AD). In particular, an Attention-based Disentanglement module is introduced to disentangle the face into Audio-face and Identity-face using speech-related facial action unit (AU) information. To precisely regulate how audio affects the talking face, we only fuse the Audio-face with audio feature. In addition, AU information is also utilized to supervise the fusion of these two modalities. Extensive qualitative and quantitative experiments demonstrate that our NeRF-AD outperforms state-of-the-art methods in generating realistic talking face videos, including image quality and lip synchronization. To view video results, please refer to https://xiaoxingliu02.github.io/NeRF-AD.

Figures (3)

• Figure 1: Overview of NeRF-AD. The gray area represents the Attention-based Disentanglement module proposed to disentangle the input talking face into Audio-face $I_{aud-face}$ and Identity-face $I_{id-face}$. Feature extraction & fusion module is used to extract the identity feature $f_{id}$ from $I_{id-face}$. And encoder $E_a$ is used to extract the Audio-face feature $f_{aud-face}$. $f_{aud-face}$ is then fused with audio feature $f_a$ extracted from DeepSpeech module. Subsequently, decoder $D$ generates feature map $\hat{I}_{fus-af}$ supervised by AU information. After that, feature encoder $E_f$ is used to generate the fused Audio-face feature vector $\hat{f}_{aud-face}$. Finally, the conditional NeRF is presented to accurately render the talking face images with $f_{id}$ and $\hat{f}_{aud-face}$ as conditions.
• Figure 2: The framework of AU-Attention.
• Figure 3: Visual results of the comparative experiments.