DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering

TL;DR

DFA-NeRF tackles personalized talking head generation by decoupling lip motion and personalized attributes. Lip motion is deterministically predicted from audio via contrastive learning, while head pose and eye-blink patterns are generated probabilistically with a Transformer GP-VAE and Gaussian Process temporal modeling, all conditioned into a dynamic NeRF. The approach yields state-of-the-art results in image quality, lip-sync accuracy, and naturalistic motion, demonstrated across multiple benchmarks and user studies. It enables high-fidelity, controllable talking-head synthesis with realistic attributes, though it highlights ethical considerations and potential limitations in multi-speaker scenarios and inference speed.

Abstract

While recent advances in deep neural networks have made it possible to render high-quality images, generating photo-realistic and personalized talking head remains challenging. With given audio, the key to tackling this task is synchronizing lip movement and simultaneously generating personalized attributes like head movement and eye blink. In this work, we observe that the input audio is highly correlated to lip motion while less correlated to other personalized attributes (e.g., head movements). Inspired by this, we propose a novel framework based on neural radiance field to pursue high-fidelity and personalized talking head generation. Specifically, neural radiance field takes lip movements features and personalized attributes as two disentangled conditions, where lip movements are directly predicted from the audio inputs to achieve lip-synchronized generation. In the meanwhile, personalized attributes are sampled from a probabilistic model, where we design a Transformer-based variational autoencoder sampled from Gaussian Process to learn plausible and natural-looking head pose and eye blink. Experiments on several benchmarks demonstrate that our method achieves significantly better results than state-of-the-art methods.

Figures (7)

• Figure 1: Illustration of Disentangled Face Attributes Neural Radiance Field (DFA-NeRF). Our framework disentangles the face attributes into head poses, eye blink features and lip motion features, which are generated from the input audio. Then we train a dynamic NeRF conditioned on these features to synthesize high-quality personalized talking head images.
• Figure 2: An overview of our proposed framework. Our method mainly consists of two parts: face attributes extraction and volume rendering. For the first part, pose and expression parameters are extracted from input videos. Then face attributes disentanglement module $F_d$ is introduced to disentangle eye blink embeddings $\mathbf{f}_e$ and lip motion embeddings $\mathbf{f}_m$. We use Transformer GP-VAE to generate personalized attributes such as head poses and eye blink features. We use a contrastive learning method for the lip motion to synchronize the audio feature $\mathbf{f}_a$ with the mouth movement feature $\mathbf{f}_m$. Afterward, in the volume rendering stage, we use generated pose $\mathbf{h}^{\prime}$ as the view direction. Meanwhile, the generated eye blink feature and synchronized audio feature are concatenated and serve as a condition $\mathbf{f}_c$ of the NeRF. Finally, we use volume rendering to render the image.
• Figure 3: Illustration of face attributes disentanglement: the mouth embedding code is switched as an example. $\mathbf{f}_e$ represents the eye blink embedding code, $\mathbf{f}_m$ denotes the mouth embedding code. The subscripts $A$ and $B$ indicate different people.
• Figure 4: a) Structure of Transformer VAE. This module will restructure the input face attribute sequences $\mathbf{h}$ and learn a Gaussian prior latent space $Z$. The encoder $E_h$ and decoder $D$ are both Transformers. b) Structure of Transformer-based cross-modal encoder. Audio and BOP sequences share a common feature space, same as the trained latent space $Z$ in (a). We use the reparameterization trick to resample new latent codes $\mathbf{z}_{ah}$ through Gaussian Process. The trained decoder $D$ decodes $\mathbf{z}_{ah}$ to predict future face attributes.
• Figure 5: Qualitative results. The top rows are the reference source videos. ATVG Hierarchical-Cross-Modal-Talking can only generate the cropped frontal talking head. LSP Live-Speech-Portraits can generate natural head movements and eye blinks, but the lip motion is not accurately synchronized with the input voice. Though MakeitTalk MakeItTalk can achieve subtle head movements and eye blinks, the mouth shapes are not accurate, which are marked with a box. The lip motion of Wav2lip prajwal2020lip is accurate, but it can only synthesize the mouth movement of the talking face. AD-NeRF adnerf use two NeRFs to generate the head, lower body separately, and we observe a clear white gap between the head part and the body part(marked with a box in the figure). Our method generates photo-realistic talking heads with diverse head movements.