AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation

TL;DR

AVI-Talking tackles the challenge of generating expressive 3D talking faces that align with the speaker's emotion and speaking state by introducing an intermediate visual instruction produced by Large Language Models from audio. The method splits audio-to-video generation into an audio-visual instruction stage and an instruction-following synthesis stage, leveraging a Q-Former for speech-to-instruction alignment and a disentangled motion prior with a diffusion prior to realize expressive, lip-synced faces. Key contributions include a soft prompting strategy for LLMs, a two-space (speech content vs. content-irrelevant) motion prior, and a diffusion-based alignment that yields diverse, emotionally coherent 3D talking faces. The approach demonstrates improved expressive realism and user-alterable control, with robust cross-dataset performance and comprehensive ablations, albeit relying on labeled instruction data and facing some lip-sync trade-offs when emphasizing expressiveness.

Abstract

While considerable progress has been made in achieving accurate lip synchronization for 3D speech-driven talking face generation, the task of incorporating expressive facial detail synthesis aligned with the speaker's speaking status remains challenging. Our goal is to directly leverage the inherent style information conveyed by human speech for generating an expressive talking face that aligns with the speaking status. In this paper, we propose AVI-Talking, an Audio-Visual Instruction system for expressive Talking face generation. This system harnesses the robust contextual reasoning and hallucination capability offered by Large Language Models (LLMs) to instruct the realistic synthesis of 3D talking faces. Instead of directly learning facial movements from human speech, our two-stage strategy involves the LLMs first comprehending audio information and generating instructions implying expressive facial details seamlessly corresponding to the speech. Subsequently, a diffusion-based generative network executes these instructions. This two-stage process, coupled with the incorporation of LLMs, enhances model interpretability and provides users with flexibility to comprehend instructions and specify desired operations or modifications. Extensive experiments showcase the effectiveness of our approach in producing vivid talking faces with expressive facial movements and consistent emotional status.

Figures (10)

• Figure 1: In contrast to previous approaches that directly learn facial motions from speaker speech, our framework introduces an audio-visual instruction module achieved by LLMs to instruct the talking face synthesis network.
• Figure 2: The overall pipeline of our Audio-Visual Instruction Talking (AVI-Talking) Framework. Given a clip of speaker speech { $a_i \}_{i=1}^{T_a}$, it is first processed by Large Language Models (LLMs) to propose visual instructions encompassing plausible facial detail descriptions. Subsequently, these visual instructions, together with audio clip, are separately fed into the talking face instruction system to generate a time series of 3D parametric coefficients $\{\theta_i,\psi_i\}_{i=1}^{T_a}$.
• Figure 3: The Q-Former architecture leverages the standard Perceiver network alayrac2022flamingo to compress speech input to a fixed-length audio embedding $\textbf{F}_{si}^{a} \in \mathcal{R}^{q_a \times l}$. A contrastive loss $\mathcal{L}_{cont}^{a2i}$ is applied to encourage the queries extract audio representation that are most relevant to visual instructions.
• Figure 4: To establish a disentangled expressive motion prior, we learn two complementary latent spaces, speech content space and content irrelevant space. In speech content space, we represent lip movements related to speech content, while in the content irrelevant space, we capture facial expressions correlated with the speaking state.
• Figure 5: Within the content irrelevant space, we contrastively align the visual instruction with style embedding to obtain a aligned feature $c$, upon which a diffusion prior network is employed to further map it towards the distribution of the pre-trained talking prior.