MMHead: Towards Fine-grained Multi-modal 3D Facial Animation

TL;DR

MMHead introduces the first large-scale multi-modal 3D facial animation dataset with rich hierarchical text annotations, combining audio and detailed text descriptions to drive 3D facial motion represented by FLAME parameters. It benchmarks two tasks: text-induced 3D talking head animation and text-to-3D facial motion generation, enabling non-deterministic, diverse motion synthesis. The authors propose MM2Face, a two-stage VQ-VAE plus autoregressive transformer framework that tokenizes motions and fuses audio and text to generate plausible, diverse motions, achieving competitive results across benchmarks. This dataset and approach are poised to accelerate research in fine-grained, text-guided multi-modal 3D facial animation and its applications in avatars and virtual agents.

Abstract

3D facial animation has attracted considerable attention due to its extensive applications in the multimedia field. Audio-driven 3D facial animation has been widely explored with promising results. However, multi-modal 3D facial animation, especially text-guided 3D facial animation is rarely explored due to the lack of multi-modal 3D facial animation dataset. To fill this gap, we first construct a large-scale multi-modal 3D facial animation dataset, MMHead, which consists of 49 hours of 3D facial motion sequences, speech audios, and rich hierarchical text annotations. Each text annotation contains abstract action and emotion descriptions, fine-grained facial and head movements (i.e., expression and head pose) descriptions, and three possible scenarios that may cause such emotion. Concretely, we integrate five public 2D portrait video datasets, and propose an automatic pipeline to 1) reconstruct 3D facial motion sequences from monocular videos; and 2) obtain hierarchical text annotations with the help of AU detection and ChatGPT. Based on the MMHead dataset, we establish benchmarks for two new tasks: text-induced 3D talking head animation and text-to-3D facial motion generation. Moreover, a simple but efficient VQ-VAE-based method named MM2Face is proposed to unify the multi-modal information and generate diverse and plausible 3D facial motions, which achieves competitive results on both benchmarks. Extensive experiments and comprehensive analysis demonstrate the significant potential of our dataset and benchmarks in promoting the development of multi-modal 3D facial animation.

Figures (16)

• Figure 1: Dataset construction pipeline. We first integrate five portrait video datasets and filter the data to obtain the candidate videos for constructing our 3D facial animation dataset. Then, high-precision 3D facial motion sequences are obtained from candidate videos through monocular 3D face reconstruction, FLAME parameter optimization, and manual screening in turn. Finally, we utilize ChatGPT with well-designed prompts to obtain abstract and fine-grained text annotations.
• Figure 2: Text annotation pipeline with well-designed prompts. We use different prompts and input information to annotate each data with five types of text descriptions separately by ChatGPT.
• Figure 3: Overview of our MM2Face framework. In stage I, we utilize a VQ-VAE $\mathcal{V}$ to tokenize the FLAME facial motions $\boldsymbol{F}_{1:T}$ to a sequence of motion tokens. Then in stage II, we utilize a causal auto-regressive transformer MM2Face $\mathcal{G}$ to generate discrete motion tokens $\boldsymbol{\Tilde{F}}$ sequentially from audio and text inputs.
• Figure 4: Qualitative results of MM2Face on benchmark I: text-induced 3D talking head animation.
• Figure 5: Qualitative results of MM2Face on benchmark II: text-to-3D facial motion generation.