DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations

TL;DR

DualTalk tackles the lack of bidirectional, multi-round interaction in 3D talking head generation by introducing a unified dual-speaker framework and a large dual-speaker dataset. It combines four modules—Dual-Speaker Joint Encoder, Cross-Modal Temporal Enhancer, Dual-Speaker Interaction Module, and Expressive Synthesis Module—to model speaking and listening behaviors with context-aware synchronization. The authors release a ~50 hour dataset with 1,052 identities and 5,858 clips to benchmark dual-speaker conversations. Experiments show that DualTalk surpasses state-of-the-art baselines in lip synchronization, listener feedback, and interaction dynamics, enabling more natural and expressive two-person conversations.

Abstract

In face-to-face conversations, individuals need to switch between speaking and listening roles seamlessly. Existing 3D talking head generation models focus solely on speaking or listening, neglecting the natural dynamics of interactive conversation, which leads to unnatural interactions and awkward transitions. To address this issue, we propose a new task -- multi-round dual-speaker interaction for 3D talking head generation -- which requires models to handle and generate both speaking and listening behaviors in continuous conversation. To solve this task, we introduce DualTalk, a novel unified framework that integrates the dynamic behaviors of speakers and listeners to simulate realistic and coherent dialogue interactions. This framework not only synthesizes lifelike talking heads when speaking but also generates continuous and vivid non-verbal feedback when listening, effectively capturing the interplay between the roles. We also create a new dataset featuring 50 hours of multi-round conversations with over 1,000 characters, where participants continuously switch between speaking and listening roles. Extensive experiments demonstrate that our method significantly enhances the naturalness and expressiveness of 3D talking heads in dual-speaker conversations. We recommend watching the supplementary video: https://ziqiaopeng.github.io/dualtalk.

Figures (4)

• Figure 1: Overview of DualTalk. DualTalk consists of four components: (a) Dual-Speaker Joint Encoder, (b) Cross-Modal Temporal Enhancer, (c) Dual-Speaker Interaction Module, and (d) Expressive Synthesis Module, enabling the generation of smooth and natural dual-speaker interactions.
• Figure 2: Dataset construction pipeline. The pipeline takes raw two-speaker videos and paired audio as input. It outputs segmented video clips, isolated audio streams for each speaker, 3D facial mesh data, and speaker round count statistics, providing high-quality, synchronized multimodal data for training.
• Figure 3: Distribution of conversation rounds in DualTalk dataset and example samples.
• Figure 4: Qualitative comparison of speaking and listening states. The left side shows facial expressions in the speaking state, with DualTalk achieving more accurate lip movements compared to other methods. The right side shows expressions in the listening state, where DualTalk captures expressive responses like smiling and nodding, outperforming other methods in naturalness and contextual relevance.