EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Head

TL;DR

This work tackles the problem of high-fidelity, emotion-controllable 3D talking head synthesis with free-view rendering. It introduces the EmoTalk3D dataset, a calibrated multi-view collection with emotion annotations and per-frame 3D geometry, and a novel Speech-to-Geometry-to-Appearance framework. The method decomposes appearance into canonical and dynamic components using $4D$ Gaussians, with S2GNet predicting $4D$ point clouds from audio and emotion cues, and G2ANet rendering dynamic facial details such as wrinkles. Experimental results show improved lip synchronization, rendering quality, and explicit emotion control across wide viewing angles, outperforming baselines, with ablations confirming the importance of geometry-guided synthesis and emotion conditioning. The dataset and code are released to support further research in emotion-aware 3D talking heads.

Abstract

We present a novel approach for synthesizing 3D talking heads with controllable emotion, featuring enhanced lip synchronization and rendering quality. Despite significant progress in the field, prior methods still suffer from multi-view consistency and a lack of emotional expressiveness. To address these issues, we collect EmoTalk3D dataset with calibrated multi-view videos, emotional annotations, and per-frame 3D geometry. By training on the EmoTalk3D dataset, we propose a \textit{`Speech-to-Geometry-to-Appearance'} mapping framework that first predicts faithful 3D geometry sequence from the audio features, then the appearance of a 3D talking head represented by 4D Gaussians is synthesized from the predicted geometry. The appearance is further disentangled into canonical and dynamic Gaussians, learned from multi-view videos, and fused to render free-view talking head animation. Moreover, our model enables controllable emotion in the generated talking heads and can be rendered in wide-range views. Our method exhibits improved rendering quality and stability in lip motion generation while capturing dynamic facial details such as wrinkles and subtle expressions. Experiments demonstrate the effectiveness of our approach in generating high-fidelity and emotion-controllable 3D talking heads. The code and EmoTalk3D dataset are released at https://nju-3dv.github.io/projects/EmoTalk3D.

Figures (7)

• Figure 1: Given a speech signal, our method can synthesize high-fidelity, emotion-controllable talking head that can be rendered over a wide range of viewing angles.
• Figure 2: EmoTalk3D Dataset. We collect a multi-view talking face dataset, where each subject's data contains $8$ emotions and the reconstructed 3D mesh model for each frame. It is worth noting that the provided per-frame 3D models cannot represent detailed 3D shapes like wrinkles but can be learned from videos by our G2ANet (Sec. \ref{['sec:g2a']}).
• Figure 3: Overall Pipeline. The pipeline consists of five modules: 1) Audio Encoder that parses content features from input speech; 2) Speech-to-Geometry Network (S2GNet) that predicts dynamic 3D point clouds from the features; 3) Gaussian Optimization and Completion Module for establishing a canonical appearance; 4) Geometry-to-Appearance Network (G2ANet) that synthesizes facial appearance based on dynamic 3D point cloud; and 5) Rendering module for rendering dynamic Gaussians into free-view animations.
• Figure 4: Points Completion. S2GNet solely generates the point cloud for the facial region. In contrast, points other than the facial region (OTF points) are optimized from uniformly initialized points. This figure illustrates the gradual optimization process of the OTF points, culminating in forming a complete head structure.
• Figure 5: Multi-view Synthesis and Emotional Control.