Uncertainty-Aware 3D Emotional Talking Face Synthesis with Emotion Prior Distillation

TL;DR

This work tackles two core problems in 3D emotional talking-face synthesis: weak audio–vision emotion alignment and suboptimal multi-view fusion that ignores uncertainty. It introduces UA-3DTalk, a 3D Gaussian Splatting–based framework with three modules—Prior Extraction, Emotion Distillation, and Uncertainty-based Deformation—to enable audio-driven, finely controlled emotional expression with adaptive multi-view fusion. Key innovations include disentangling audio into $f_{exp}$ and $f_{tone}$ for synchronization and identity, a multi-modal emotion distillation pipeline with $4$-D Gaussian encoding over multi-resolution code-books for precise micro-expressions, and a principled uncertainty model that jointly estimates aleatoric and epistemic uncertainty to drive fusion and per-primitive deformation. Across regular and emotional datasets, UA-3DTalk achieves state-of-the-art performance with improvements in $E ext{-}FID$, SyncC, and LPIPS, demonstrating robust, audio-driven rendering without reliance on reference videos or predefined 3DMM parameters.

Abstract

Emotional Talking Face synthesis is pivotal in multimedia and signal processing, yet existing 3D methods suffer from two critical challenges: poor audio-vision emotion alignment, manifested as difficult audio emotion extraction and inadequate control over emotional micro-expressions; and a one-size-fits-all multi-view fusion strategy that overlooks uncertainty and feature quality differences, undermining rendering quality. We propose UA-3DTalk, Uncertainty-Aware 3D Emotional Talking Face Synthesis with emotion prior distillation, which has three core modules: the Prior Extraction module disentangles audio into content-synchronized features for alignment and person-specific complementary features for individualization; the Emotion Distillation module introduces a multi-modal attention-weighted fusion mechanism and 4D Gaussian encoding with multi-resolution code-books, enabling fine-grained audio emotion extraction and precise control of emotional micro-expressions; the Uncertainty-based Deformation deploys uncertainty blocks to estimate view-specific aleatoric (input noise) and epistemic (model parameters) uncertainty, realizing adaptive multi-view fusion and incorporating a multi-head decoder for Gaussian primitive optimization to mitigate the limitations of uniform-weight fusion. Extensive experiments on regular and emotional datasets show UA-3DTalk outperforms state-of-the-art methods like DEGSTalk and EDTalk by 5.2% in E-FID for emotion alignment, 3.1% in SyncC for lip synchronization, and 0.015 in LPIPS for rendering quality. Project page: https://mrask999.github.io/UA-3DTalk

Figures (3)

• Figure 1: Overall architecture of the proposed UA-3DTalk framework: (a) End-to-end pipeline of the face branch, integrating Prior Extraction, Emotion Distillation, and Uncertainty-based Deformation for 3D Gaussian primitive adjustment and rendering; (b) Detailed framework of the Prior Extraction module; (c) Technical pipeline of the Emotion Distillation module.
• Figure 2: Uncertainty-Based Deformation. (Left) Uncertainty-aware multi-view feature fusion module; (Right) Multi-head Gaussian deformation decoder.
• Figure 3: Visual results of the comparative experiments.