Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expert

TL;DR

This work tackles the challenge of intelligible lip movements in speech-driven 3D facial animation by introducing an audio-visual perceptual loss that leverages a lip-reading expert trained on large-scale 2D talking-face data. The framework combines a 3D facial animator with a speech-informed lip reading expert, connected through a differentiable renderer to align lip motion with spoken content; training proceeds in two stages to transfer priors from 2D to 3D domains. The AV loss integrates a joint CTC/attention objective and a regularizing lip-vertex term, yielding improved lip synchronization (LVE) and lip readability (CER/VER) on BIWI and VOCASET across two strong baselines (FaceFormer and CodeTalker). This approach enhances realism and intelligibility of lip motions, reducing manual post-editing in production pipelines and offering a practical method for audio-visual-guided 3D talking-head synthesis, with code publicly available. $\mathcal{L} = \lambda_{\rm mse}{\mathcal L}_{\rm mse} + \lambda_{\rm av}{\mathcal L}_{\rm av} + \lambda_{\rm rlv}{\mathcal L}_{\rm rlv}$, where ${\mathcal L}_{\rm av} = \lambda_{\rm ctc}{\mathcal L}_{\rm ctc} + \lambda_{\rm ce}{\mathcal L}_{\rm ce}$.

Abstract

Speech-driven 3D facial animation has recently garnered attention due to its cost-effective usability in multimedia production. However, most current advances overlook the intelligibility of lip movements, limiting the realism of facial expressions. In this paper, we introduce a method for speech-driven 3D facial animation to generate accurate lip movements, proposing an audio-visual multimodal perceptual loss. This loss provides guidance to train the speech-driven 3D facial animators to generate plausible lip motions aligned with the spoken transcripts. Furthermore, to incorporate the proposed audio-visual perceptual loss, we devise an audio-visual lip reading expert leveraging its prior knowledge about correlations between speech and lip motions. We validate the effectiveness of our approach through broad experiments, showing noticeable improvements in lip synchronization and lip readability performance. Codes are available at https://3d-talking-head-avguide.github.io/.

Figures (3)

• Figure 1: Overview of our proposed framework. We adopt the audio-visual lip reading expert autoavsr trained on the large-scale 2D datasets lrs2lrs3avspeechvoxceleb2 and finetune it on 3D datasets biwivoca concurrently with training the 3D facial animator. Given an input speech signal, a 3D facial animator regresses a sequence of 3D facial meshes and the following lip reading expert predicts the spoken transcript considering both the input speech signal and the sequence of lip regions of output faces.
• Figure 2: Qualitative comparisons of output facial movements on VOCASET and BIWI. Compared to the 3D facial animator baselines faceformercodetalker, the outputs of our method show better lip synchronization quality for both lip closure and opening words.
• Figure 3: t-SNE visualization for features of audio-visual/visual-only lip reading expert. Distinct separation of features for words "just" (red) and "must" (blue) is observed in the audio-visual lip reading expert. However, with visual-only input, features become entangled, hindering distinction.