A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation

TL;DR

The paper addresses the challenge of aligning audio with both lip and head dynamics in talking head generation by introducing a multi-scale audio-visual synchrony framework. It combines a pyramid of syncers trained on multi-scale representations with a multi-scale autoregressive generator (MS-Sync) to capture short- and long-horizon motion correlations in a low-dimensional 60D keypoint space, then reconstructs video via a reenactment model. Key contributions include the multi-scale AV synchrony loss, the four-branch multi-scale generator with a transformer backbone, and extensive cross-dataset experiments showing improved AV synchrony and head-motion realism. This approach enhances the realism and consistency of audio-driven facial animation and is readily adaptable to other AV generation tasks that require multi-scale temporal coherence.

Abstract

Animating still face images with deep generative models using a speech input signal is an active research topic and has seen important recent progress.However, much of the effort has been put into lip syncing and rendering quality while the generation of natural head motion, let alone the audio-visual correlation between head motion and speech, has often been neglected.In this work, we propose a multi-scale audio-visual synchrony loss and a multi-scale autoregressive GAN to better handle short and long-term correlation between speech and the dynamics of the head and lips.In particular, we train a stack of syncer models on multimodal input pyramids and use these models as guidance in a multi-scale generator network to produce audio-aligned motion unfolding over diverse time scales.Both the pyramid of audio-visual syncers and the generative models are trained in a low-dimensional space that fully preserves dynamics cues.The experiments show significant improvements over the state-of-the-art in head motion dynamics quality and especially in multi-scale audio-visual synchrony on a collection of benchmark datasets.

Figures (7)

• Figure 1: Given a source image and a driving audio signal, our model generates a talking head video sequence correlated with the input speech over multiple time scales, ensuring accurate lips and head dynamics.
• Figure 2: Left. A stack of syncer networks $S^i$ are trained on multi-scale positive and negative multimodal pairs using contrastive losses. Right. The syncer models are frozen and used to compute the multi-scale audio-visual synchrony loss of the generative model.
• Figure 3: Left. Our network is composed of an autoregressive Transformer temporal module and a convolutional multi-scale module. Right. Details of the multi-scale module.
• Figure 4: Comparison between different one-shot talking head generation methods with explicit head motion treatment on hard samples: left with a widely open initial mouth, right with closed eyes. Consecutive frames are 600ms apart. MS-Sync can deal with these samples while producing rigid head motion synchronized with the speech.
• Figure 5: Comparison between different one-shot talking head generation methods with explicit head motion treatment. Consecutive frames are 600ms apart.