Inter-Diffusion Generation Model of Speakers and Listeners for Effective Communication

TL;DR

This work tackles the challenge of generating natural, context-aware gestures for both speakers and listeners in interactive communication. It introduces an Inter-Diffusion Generation Model with a dual-branch, inter-diffusion framework that conditions speaker gestures on speech while generating listener responses, trained under the diffusion objective with classifier-free guidance. Evaluations on the combined TWH and ZeroEGG datasets show superior naturalness, beat alignment, and diversity (FGD, BA, DIV), and user studies confirm improved human-likeness and listener coherence. The approach enables more immersive virtual communication with broad applications in VR, gaming, and film, while noting limitations in generation speed and data requirements that point to future improvements via motion capture and dataset expansion.

Abstract

Full-body gestures play a pivotal role in natural interactions and are crucial for achieving effective communication. Nevertheless, most existing studies primarily focus on the gesture generation of speakers, overlooking the vital role of listeners in the interaction process and failing to fully explore the dynamic interaction between them. This paper innovatively proposes an Inter-Diffusion Generation Model of Speakers and Listeners for Effective Communication. For the first time, we integrate the full-body gestures of listeners into the generation framework. By devising a novel inter-diffusion mechanism, this model can accurately capture the complex interaction patterns between speakers and listeners during communication. In the model construction process, based on the advanced diffusion model architecture, we innovatively introduce interaction conditions and the GAN model to increase the denoising step size. As a result, when generating gesture sequences, the model can not only dynamically generate based on the speaker's speech information but also respond in realtime to the listener's feedback, enabling synergistic interaction between the two. Abundant experimental results demonstrate that compared with the current state-of-the-art gesture generation methods, the model we proposed has achieved remarkable improvements in the naturalness, coherence, and speech-gesture synchronization of the generated gestures. In the subjective evaluation experiments, users highly praised the generated interaction scenarios, believing that they are closer to real life human communication situations. Objective index evaluations also show that our model outperforms the baseline methods in multiple key indicators, providing more powerful support for effective communication.

Figures (2)

• Figure 1: Overview training process of our Inter-Diffusion.
• Figure 2: In comparison to contemporary diffusion-based methods such as DSG and FreeTalker, our approach achieves the best result.