Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style

TL;DR

This paper presents an innovative audio-driven talking face generation method called Style2Talker, which outperforms existing state-of-the-art methods in terms of audio-lip synchronization and performance of both emotion style and art style.

Abstract

Although automatically animating audio-driven talking heads has recently received growing interest, previous efforts have mainly concentrated on achieving lip synchronization with the audio, neglecting two crucial elements for generating expressive videos: emotion style and art style. In this paper, we present an innovative audio-driven talking face generation method called Style2Talker. It involves two stylized stages, namely Style-E and Style-A, which integrate text-controlled emotion style and picture-controlled art style into the final output. In order to prepare the scarce emotional text descriptions corresponding to the videos, we propose a labor-free paradigm that employs large-scale pretrained models to automatically annotate emotional text labels for existing audiovisual datasets. Incorporating the synthetic emotion texts, the Style-E stage utilizes a large-scale CLIP model to extract emotion representations, which are combined with the audio, serving as the condition for an efficient latent diffusion model designed to produce emotional motion coefficients of a 3DMM model. Moving on to the Style-A stage, we develop a coefficient-driven motion generator and an art-specific style path embedded in the well-known StyleGAN. This allows us to synthesize high-resolution artistically stylized talking head videos using the generated emotional motion coefficients and an art style source picture. Moreover, to better preserve image details and avoid artifacts, we provide StyleGAN with the multi-scale content features extracted from the identity image and refine its intermediate feature maps by the designed content encoder and refinement network, respectively. Extensive experimental results demonstrate our method outperforms existing state-of-the-art methods in terms of audio-lip synchronization and performance of both emotion style and art style.

Figures (5)

• Figure 1: Illustrative animations produced by our Style$^2$Talker. Our approach takes an identity image and an audio clip as inputs and generates a talking head with emotion style and art style, which are controlled respectively by an emotion source text and an art source picture.
• Figure 2: The overview of the proposed $\text{Style}^2\text{Talker}$. (a) Style-E: Emotion Style Transfer. In the diffusion process, we start by extracting the 3D expression coefficient sequence $\mathbf{\beta}^{1:N}_{0}$ from the ground truth video $y^{1:N}$. Then, we iteratively add Gaussian noise by $q\left(\mathbf{\beta}^{1:N}_\mathbf{t} \mid \mathbf{\beta}^{1:N}_{\mathbf{t-1}}\right) :=\mathcal{N}\left(\mathbf{\beta}^{1:N}_{\mathbf{t-1}} ; \sqrt{\alpha_\mathbf{t}} \mathbf{\beta}^{1:N}_{\mathbf{t-1}}, (1-\alpha_\mathbf{t}) \mathbf{I}\right)$. A simple MLP-based denoising network $\mu_\theta$ is trained to denoise the noisy parameters $\mathbf{\beta}_\mathbf{t}$ at time $\mathbf{t}$ based on the conditioning signal $\mathbf{c}$, and $\mathbf{c}$ comprises text $\mathcal{T}$, identity image $\mathcal{I}$ and audio $\mathcal{A}$. (b) Style-A: Art Style Transfer. We employ a pre-trained encoder $E_s$ from pSp richardson2021encoding to embed the identity image $\mathcal{I}$ and art source image $\mathcal{I}_s$ to the latent code $w_i$ and $w_s$, which are fed into a ModRes to merge the style code. To alleviate content loss caused by the pSp richardson2021encoding, we introduce another Content Encoder $E_c$ to extract multi-level content features. These features are then fed into StyleGAN $G$ through skip connections and combined with style codes, serving as the input of $G$. To enable continuous frames generation, the produced emotional coefficients $\hat{\mathbf{\beta}}^{1:N}_{0}$ are converted to flow field map $f$, which in turn warps feature map $m$ of StyleGAN to $\hat{m}$, achieving talking head generation with style$^2$.
• Figure 3: Qualitative comparisons with state-of-the-art methods. The input for our Style$^2$Talker and SOTAs are marked by '*' and '$\#$', respectively. We keep the original color style for better comparison.
• Figure 4: User study results.
• Figure 5: Visualization Results of ablation study.