Talk3D: High-Fidelity Talking Portrait Synthesis via Personalized 3D Generative Prior

TL;DR

Talk3D tackles the challenge of high-fidelity, pose-controlled talking head synthesis from monocular video by leveraging a personalized 3D-aware generative prior and NeRF-space editing. It introduces an audio-guided attention U-Net that predicts a deltaplane to deform a identity-specific triplane, enabling accurate lip-sync and 3D-consistent rendering under unseen viewpoints. The model employs region-aware conditioning tokens and split-convolution to disentangle background and torso motion from facial dynamics, and uses a comprehensive loss with a sync term to improve lip synchronization. Extensive experiments show Talk3D outperforms state-of-the-art NeRF-based and 2D methods in both quantitative metrics and qualitative assessments, with a user study confirming superior perceived realism and lip-sync. The work advances digital humans by enabling robust, view-consistent talking portraits with flexible identity editing and potential applications in virtual avatars and video conferencing, while acknowledging limitations in non-photorealistic domains and the dependency on GAN inversion workflows.

Abstract

Recent methods for audio-driven talking head synthesis often optimize neural radiance fields (NeRF) on a monocular talking portrait video, leveraging its capability to render high-fidelity and 3D-consistent novel-view frames. However, they often struggle to reconstruct complete face geometry due to the absence of comprehensive 3D information in the input monocular videos. In this paper, we introduce a novel audio-driven talking head synthesis framework, called Talk3D, that can faithfully reconstruct its plausible facial geometries by effectively adopting the pre-trained 3D-aware generative prior. Given the personalized 3D generative model, we present a novel audio-guided attention U-Net architecture that predicts the dynamic face variations in the NeRF space driven by audio. Furthermore, our model is further modulated by audio-unrelated conditioning tokens which effectively disentangle variations unrelated to audio features. Compared to existing methods, our method excels in generating realistic facial geometries even under extreme head poses. We also conduct extensive experiments showing our approach surpasses state-of-the-art benchmarks in terms of both quantitative and qualitative evaluations.

Figures (15)

• Figure 1: Visualizations of generated talking heads by state-of-the-art NeRF-based method, ER-NeRF li2023ernerf, and Talk3D rendered at extreme novel camera poses. Our Talk3D shows the robustness in generating high-fidelity realistic geometry of talking heads even at unseen poses during training.
• Figure 2: Overview of our Talk3D framework. Our model mainly utilizes a personalized generator. Given identity triplane $\mathbf{P}_\mathrm{ID}$ and $n$-th frame's conditioning tokens $\mathrm{t}_n$, audio-guided attention U-Net predicts the deltaplane $\Delta \mathbf{P}_n$ which represents the dynamic residual scene variation of the corresponding ground-truth image $I_n$. This is further combined with $\mathbf{P}_\mathrm{ID}$ through summation, forming $\mathbf{P}'_n$, and fed to the neural renderer with given camera viewpoint $\pi_n$ to generate final output image $I'_n$.
• Figure 3: Visualization of synthesized portraits from head poses unseen during training. We show a randomly selected frame from synthesized talking portraits using different rendered at different yaw and pitch ($\mathbf{y}$, $\mathbf{p}$) angles. Our method demonstrates its robustness on rendering facial images at large head angles which are rarely shown in the training video.
• Figure 4: The comparison of the keyframes and details of generated portraits. We show visualizations of our method and previous methods under the self-driven setting (left side) and the cross-driven setting (right side). Best viewed in zoom.
• Figure 5: Visualization of synthesized portraits and depth map rendered from novel viewpoints. We show a randomly selected frame from synthesized talking portraits (odd rows) and corresponding depth information (even rows) using different rendering viewpoints of yaw and pitch angles ($\mathbf{y}$, $\mathbf{p}$) with $15^{\circ}$, $10^{\circ}$ intervals.