StyleTalker: One-shot Style-based Audio-driven Talking Head Video Generation

TL;DR

StyleTalker tackles the problem of generating high-fidelity talking head videos from a single reference image and input audio by learning latent representations for identity, lip movements, and motions. It combines a StyleGAN3-based generator with GAN inversion, a contrastive lip-sync discriminator, a conditional sequential VAE for motion latent modeling, and an autoregressive prior augmented with normalizing flow to capture multi-modal audio-to-motion mappings. The framework supports two inference modes: motion-controllable generation using a separate motion source and audio-driven generation by inferring motions from audio, both while preserving identity through latent-code manipulation of the reference image. Experimental results on VoxCeleb2 show state-of-the-art lip synchronization, natural motion, and identity preservation, with ablations confirming the benefits of the normalizing flow and the lip-sync discriminator for realism and fidelity.

Abstract

We propose StyleTalker, a novel audio-driven talking head generation model that can synthesize a video of a talking person from a single reference image with accurately audio-synced lip shapes, realistic head poses, and eye blinks. Specifically, by leveraging a pretrained image generator and an image encoder, we estimate the latent codes of the talking head video that faithfully reflects the given audio. This is made possible with several newly devised components: 1) A contrastive lip-sync discriminator for accurate lip synchronization, 2) A conditional sequential variational autoencoder that learns the latent motion space disentangled from the lip movements, such that we can independently manipulate the motions and lip movements while preserving the identity. 3) An auto-regressive prior augmented with normalizing flow to learn a complex audio-to-motion multi-modal latent space. Equipped with these components, StyleTalker can generate talking head videos not only in a motion-controllable way when another motion source video is given but also in a completely audio-driven manner by inferring realistic motions from the input audio. Through extensive experiments and user studies, we show that our model is able to synthesize talking head videos with impressive perceptual quality which are accurately lip-synced with the input audios, largely outperforming state-of-the-art baselines.

Figures (9)

• Figure 1: Concept. StyleTalker generates a realistic talking head video from a single reference image and input audio in two different ways. Left: Audio-driven Generation Motions are predicted from the audio. Right: Motion-controllable Generation Motions are controlled by another motion source video.
• Figure 2: The overall training framework of StyleTalker. 1) Video & Audio driven Style Code Manipulation: Given a video, StyleTalker maps audio into audio features $a_{0:T}$ and frames into motion latent variables $m_{0:T}$. With $a_{0:T}$ and $m_{0:T}$, we manipulate the style latent code of the reference frame $\mathbf{\mathcal{W}}+_{ref}$ to generate $\widehat{\mathbf{\mathcal{W}}}+_{t:t+T_w}$. 2)Video Generation: $\widehat{\mathbf{\mathcal{W}}}+_{t:t+T_w}$ are forwarded to generate $n$ sequential frames, which are the reconstruction of the given video. 3)Audio-driven Motion Generation: We model our prior network of $m_{0:T}$ with given audio and an image as inputs. Note that $Enc_I$, $G_I$, and $D_{sync}$ are fixed during training.
• Figure 3: Contrastive Lip-sync Discriminator
• Figure 4: $\bf w$ manipulation module.
• Figure 5: The qualitative comparison of the audio-driven talking head generation performance on VoxCeleb2. The first row (yellow box) shows the frames corresponding to the given audio. The single image input (red box) is a reference image of the target identity. The Supplementary Materials contain the actual videos.