Audio-Driven Talking Face Generation with Blink Embedding and Hash Grid Landmarks Encoding

TL;DR

This study tackles realistic audio-driven talking-face synthesis by advancing NeRF-based rendering with a dynamic landmark approach. It introduces a Dynamic Landmark Transformer that maps audio features to 3D facial landmarks, augmented by blink embedding and an eye-movement module to enhance realism and reduce uncanny valley effects. Rendering efficiency is boosted using hash-grid encoding on a tri-plane NeRF, enabling faster, high-fidelity synthesis with a two-stage coarse-to-fine training regime that emphasizes mouth region details. Experimental results on Macron and Obama videos show improved lip synchronization and facial detail compared with state-of-the-art baselines, highlighting the method's potential for real-time, high-quality digital humans while acknowledging current limitations and ethical considerations.

Abstract

Dynamic Neural Radiance Fields (NeRF) have demonstrated considerable success in generating high-fidelity 3D models of talking portraits. Despite significant advancements in the rendering speed and generation quality, challenges persist in accurately and efficiently capturing mouth movements in talking portraits. To tackle this challenge, we propose an automatic method based on blink embedding and hash grid landmarks encoding in this study, which can substantially enhance the fidelity of talking faces. Specifically, we leverage facial features encoded as conditional features and integrate audio features as residual terms into our model through a Dynamic Landmark Transformer. Furthermore, we employ neural radiance fields to model the entire face, resulting in a lifelike face representation. Experimental evaluations have validated the superiority of our approach to existing methods.

Figures (8)

• Figure 1: network architecture Our model employs tri-plana hashing to represent facial regions and a Dynamic landmark Transformer to process audio features as queries to retrieve coordinates and corresponding attributes within NeRF space. Subsequently, we process the volumetric density (denoted as (c, $\sigma$)) of the facial representation from NeRF space by volume rendering to form a facial video of the speaking head.
• Figure 2: The mapping network comprising convolutional layers and fully connected layers, utilizing only six landmarks surrounding the eyes. This network takes as input an eye-related vector and learns the current size of the eyes, which is then fed into a control module. The control module predicts the movement of relevant landmarks for eye transformation.
• Figure 3: Illustration of the key frames and detailed comparisons of the generated portraits. Our method demonstrates significant advantages in overall visual coherence and lip synchronization on the Macron dataset.
• Figure 4: Comparisons on the collected videos of Obama. Our method exhibits considerable advantages in mouth details, eye dynamics, and artifact reduction.
• Figure 5: network architecture Our model employs tri-plana hashing to represent facial regions and a Dynamic landmark Transformer to process audio features as queries to retrieve coordinates and corresponding attributes within NeRF space. Subsequently, we process the volumetric density (denoted as (c, $\sigma$)) of the facial representation from NeRF space by volume rendering to form a facial video of the speaking head.