TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation

TL;DR

TAVID tackles the challenge of simultaneous, text-driven generation of interactive video and conversational speech by introducing two cross-modal mappers, the Motion Mapper and the Speaker Mapper, that enable bidirectional information exchange between audio and visual streams. The framework conditions both video and speech pipelines on multi-stream semantic tokens and leverages a diffusion-based video generator along with an encoder–decoder speech model to produce synchronized, identity-consistent outputs. Comprehensive experiments across interactive head generation, single-role talking/listening, and face-stylized speech demonstrate superior visual realism, lip-sync, turn-taking fluency, and speech naturalness, with ablations validating the essential role of prosody, cross-modal coupling, and identity-aligned speaker representations. By unifying text-driven content, visual identity, and conversational dynamics, TAVID offers a scalable path toward human-like audiovisual agents for tutoring, companionship, and social robotics.

Abstract

The objective of this paper is to jointly synthesize interactive videos and conversational speech from text and reference images. With the ultimate goal of building human-like conversational systems, recent studies have explored talking or listening head generation as well as conversational speech generation. However, these works are typically studied in isolation, overlooking the multimodal nature of human conversation, which involves tightly coupled audio-visual interactions. In this paper, we introduce TAVID, a unified framework that generates both interactive faces and conversational speech in a synchronized manner. TAVID integrates face and speech generation pipelines through two cross-modal mappers (i.e., a motion mapper and a speaker mapper), which enable bidirectional exchange of complementary information between the audio and visual modalities. We evaluate our system across four dimensions: talking face realism, listening head responsiveness, dyadic interaction fluency, and speech quality. Extensive experiments demonstrate the effectiveness of our approach across all these aspects.

Figures (5)

• Figure 1: Overview of TAVID framework. Given a text dialogue and reference images, TAVID simultaneously produces interactive videos and conversational speech with natural turn-taking, accurate synchronization and expressive facial dynamics.
• Figure 2: The overall architecture of TAVID. Given a text dialogue and a reference image, our approach generates both interactive video and conversational speech, guided by two cross-modal mappers. The Motion Mapper predicts interactive motions from multi-stream semantic tokens, while the Speaker Mapper models vocal characteristics from the reference image. The detailed architectures of the cross-modal mappers are shown on the right side.
• Figure 3: Analysis of integration strategies for the Motion Mapper. $c_\text{mot}$ denotes the interactive motion features extracted from multi-stream semantic tokens, whereas $c_\text{mot}^{1}$ and $c_\text{mot}^{2}$ correspond to motion features specific to each individual stream.
• Figure 4: Qualitative results of TAVID on the Seamless Interaction test set. Our method converts text inputs into realistic conversational speech and interactive video, capturing precise lip motions, responsive behaviors, and speech overlaps.
• Figure 5: AB preference results of the Speaker Mapper for visual and acoustic identity matching.