MANGO:Natural Multi-speaker 3D Talking Head Generation via 2D-Lifted Enhancement

TL;DR

This work tackles the challenge of natural, bidirectional multi-speaker 3D talking head generation, addressing the shortcomings of single-speaker models and reliance on noisy pseudo-3D labels. It introduces MANGO, a two-stage framework where a diffusion-based motion generator conditioned on dual-audio inputs yields 3D facial motion, followed by a fast 3D Gaussian Renderer that produces high-fidelity 2D frames under 2D supervision; the two stages are pretrained separately and then jointly trained. A key contribution is the MANGO-Dialog dataset, providing over 50 hours of aligned 2D-3D dialogue data across 500 identities to support multi-speaker learning. Quantitative and qualitative evaluations on MANGO-Dialog and the DualTalk baseline demonstrate superior 3D mesh accuracy and 2D image fidelity, validating the effectiveness of the dual-audio fusion mechanism and image-level supervision in achieving 2D-lifted, 3D-consistent talking heads with natural listening and speaking dynamics. The approach holds promise for more realistic and controllable digital humans in interactive settings, while also motivating future work on broader emotions and faster inference.

Abstract

Current audio-driven 3D head generation methods mainly focus on single-speaker scenarios, lacking natural, bidirectional listen-and-speak interaction. Achieving seamless conversational behavior, where speaking and listening states transition fluidly remains a key challenge. Existing 3D conversational avatar approaches rely on error-prone pseudo-3D labels that fail to capture fine-grained facial dynamics. To address these limitations, we introduce a novel two-stage framework MANGO, which leveraging pure image-level supervision by alternately training to mitigate the noise introduced by pseudo-3D labels, thereby achieving better alignment with real-world conversational behaviors. Specifically, in the first stage, a diffusion-based transformer with a dual-audio interaction module models natural 3D motion from multi-speaker audio. In the second stage, we use a fast 3D Gaussian Renderer to generate high-fidelity images and provide 2D-level photometric supervision for the 3D motions through alternate training. Additionally, we introduce MANGO-Dialog, a high-quality dataset with over 50 hours of aligned 2D-3D conversational data across 500+ identities. Extensive experiments demonstrate that our method achieves exceptional accuracy and realism in modeling two-person 3D dialogue motion, significantly advancing the fidelity and controllability of audio-driven talking heads.

Figures (11)

• Figure 1: The illustration shows A and B conversing, the blue box with 'L' indicates that this person is listening, while the orange box with 'T' indicates that this person is talking. The 3D mesh sequences as well as 2D video of A can be synthesized from their conversational audio and a reference image of A, with the same process applying to B.
• Figure 2: Limitations of existing 3D face reconstruction for training conversational talking head. The estimated 3D data either exhibits over-smoothed mouth movements (orange dashed curve) compared to actual lip movements (red curve), or shows exaggerated and noisy movements (blue dotted curve). Such visual misalignments are illustrated on the right. The red curve is derived by calculating the distance between the corresponding key points of the manually annotated upper and lower lips, which effectively describes the actual mouth movement. The blue curve represents the average distance between the corresponding key points of the upper and lower lips, calculated after projecting the 3D mesh reconstructed by Spectre back into 2D space. The orange curve shows the results from Teaser.
• Figure 3: The overall pipeline of MANGO. Our method first generates 3D facial motions from speech via DIM-FMM, and then synthesizes images with MG-Renderer, where 2D supervision further refines the 3D motion.
• Figure 4: Dual-audio interactive module (DIM) and fused audio motion generation module (FMM). The speech signals $\bfseries\sffamily{A}_{self}$ and $\bfseries\sffamily{A}_{other}$ are fed into Hubert for semantic features $\mathbf{h}^s_{0:w}$ and $\mathbf{h}^o_{0:w}$. These features are concatenated and fed into a multi-head self-attention module, which is then combined with $\mathbf{h}^s_{0:w}$ via residual connection, followed by concatenation with the agent speaker indicator $\mathbb{I}$. These features are then fed into FMM for motion $\hat{\mathbf{X}}_{0:w}$ generation.
• Figure 5: Visual comparison of the 3D conversational talking head generation results with SOTA methods on our MANGO-Dialog testset.