Content and Style Aware Audio-Driven Facial Animation

TL;DR

The paper addresses controllable audio-driven facial animation by explicitly separating content and style. It introduces a two-stage pipeline where Stage I learns style from high-resource audio through Mel-spectrum reconstruction and Wav2Vec2-based encoders, enabling robust style representations, while Stage II fine-tunes for 3D head mesh generation using limited data, with a non-autoregressive design and specialized losses to ensure realistic mouth motion. Key contributions include disentangled content-style representations, a Laplacian-mod regularization to suppress high-frequency artifacts, mouth-focused weighting to emphasize audio-articulated regions, and the ability to perform style transfer and content editing without retraining on full datasets. The approach reduces data requirements, improves lip-sync and articulation quality, enables mix-and-match content and style, and offers practical benefits for avatar animation and content editing, albeit with dependencies on external transcript alignment and potential ethical considerations.

Abstract

Audio-driven 3D facial animation has several virtual humans applications for content creation and editing. While several existing methods provide solutions for speech-driven animation, precise control over content (what) and style (how) of the final performance is still challenging. We propose a novel approach that takes as input an audio, and the corresponding text to extract temporally-aligned content and disentangled style representations, in order to provide controls over 3D facial animation. Our method is trained in two stages, that evolves from audio prominent styles (how it sounds) to visual prominent styles (how it looks). We leverage a high-resource audio dataset in stage I to learn styles that control speech generation in a self-supervised learning framework, and then fine-tune this model with low-resource audio/3D mesh pairs in stage II to control 3D vertex generation. We employ a non-autoregressive seq2seq formulation to model sentence-level dependencies, and better mouth articulations. Our method provides flexibility that the style of a reference audio and the content of a source audio can be combined to enable audio style transfer. Similarly, the content can be modified, e.g. muting or swapping words, that enables style-preserving content editing.

Figures (13)

• Figure 1: System Overview. During Stage I, styles are learnt via Mel-spectrum reconstruction. Stage II fine-tunes the network for the highly correlated audio-driven animation task. Left shows the architectural details of the variance adapter, and right the style encoder that outputs disentangled styles, initialised with two pre-trained Wav2Vec2 baevski2020wav2vec networks.
• Figure 2: The T-SNE plot of the style vectors applied onto ESD dataset after Stage I training over (a) six randomly chosen subjects, and (b) one subject with different emotions. (c) The method min2021meta cannot cluster well speakers, while (d) wan2020generalized does not cover other attributes.
• Figure 3: The learned styles evolve from Stage I to Stage II with different modality focuses.
• Figure 4: Ablation of the Laplacian-Mod (Lap-Mod) loss v.s. ground-truth, without (w/o) regularisation and with two levels of Tikhonov (Tik) regularisation. Red color represents errors.
• Figure 5: Mouth closures illustration, when the person is uttering "our experiment's positive outcome". The highlighted bilabial phonemes relate to four mouth closures.