EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars

TL;DR

EMOPortraits addresses the challenge of one-shot, emotion-rich head avatar synthesis in cross-driving scenarios by refining MegaPortraits with a smaller, more expressive latent space, a self-supervised latent-space regularization, and a novel canonical-volume decoupling. It also introduces a speech-driving mode that disentangles expression from pose and leverages mouth-focused PCA components to enable audio-driven lip movements and natural head rotations, alongside a new FEED dataset capturing extreme, asymmetric expressions. Empirical results show state-of-the-art performance in intense expression transfer and competitive audio-driven animation, supported by comprehensive ablations and comparisons to strong baselines. While body/shoulder motion remains outside the scope, EMOPortraits demonstrates practical impact for multimedia, virtual assistants, and mixed-reality applications, with FEED providing a valuable resource for future research in high-fidelity, multi-view facial dynamics.

Abstract

Head avatars animated by visual signals have gained popularity, particularly in cross-driving synthesis where the driver differs from the animated character, a challenging but highly practical approach. The recently presented MegaPortraits model has demonstrated state-of-the-art results in this domain. We conduct a deep examination and evaluation of this model, with a particular focus on its latent space for facial expression descriptors, and uncover several limitations with its ability to express intense face motions. To address these limitations, we propose substantial changes in both training pipeline and model architecture, to introduce our EMOPortraits model, where we: Enhance the model's capability to faithfully support intense, asymmetric face expressions, setting a new state-of-the-art result in the emotion transfer task, surpassing previous methods in both metrics and quality. Incorporate speech-driven mode to our model, achieving top-tier performance in audio-driven facial animation, making it possible to drive source identity through diverse modalities, including visual signal, audio, or a blend of both. We propose a novel multi-view video dataset featuring a wide range of intense and asymmetric facial expressions, filling the gap with absence of such data in existing datasets.

Figures (17)

• Figure 1: Selected animation results for image-driven mode of our method.
• Figure 2: Illustration of the problem in publicly available face expression data and selected examples from our FEED dataset.
• Figure 3: Comparison of latent spaces. Left plot shows that our model's latent space is wider and exhibits more even variance distribution. Also, as shown on the right plot, a greater number of principal components are involved in capturing variance across various thresholds. This implies a more robust representational capacity of expression space compared to drobyshev2023megaportraits. The VoxCeleb2 test set was used for both plots.
• Figure 4: Method Overview. We use $\mathbf{E}_\text{app}$ to extract volume features $\mathbf{V}_s$ and a global descriptor $\mathbf{e}_s$ from the source image. Then $\mathbf{E}_\text{motion}$ or $\mathbf{E}_\text{audio}$ generates motion representations from source and driver, including head rotations $\mathbf{R}_{s/d}$, translations $\mathbf{t}_{s/d}$, and expression descriptors $\mathbf{z}_{s/d}$. Using them, we predict warpings $\mathbf{w}_{s \rightarrow}$ and $\mathbf{w}_{\rightarrow d}$. First warping and $\mathbf{G}_\text{3D}$ transform $\mathbf{V}_s$ into a canonical volume $\mathbf{V}_s^C$ by removing the source motions. Second warping and $\mathbf{G}_\text{2D}$ imposes the driver's motions and renders the final image.
• Figure 5: Canonical volumes ($\textbf{V}^C$) in MegaPortraits drobyshev2023megaportraits are not expression-neutral. To show it we use three portraits: A (regular expression), B (intense expression), D (regular expression, new identity). Case 1 visualizes poor results in transferring B's intense expression to A using a self-driving mode, contrasting with the effective reconstruction of B when used as both driver and source in case 3. This discrepancy, indicative of expression leakage into the canonical volume, is further quantified by a 43% relative difference in $\textbf{V}^C$s between B and C, contrary to expectations of their similarity for same identity. In cross-driving generation case 2, using B as the driver and D as the source yielded poor results. However, in case 4, after using additive operations on source canonical volumes, the output expression is much closer to driver than in case 2. This manipulation again confirms the significant retention of expression information in canonical volumes.