Conversational Co-Speech Gesture Generation via Modeling Dialog Intention, Emotion, and Context with Diffusion Models

TL;DR

This paper addresses the challenge of generating co-speech gestures for multi-person conversations by introducing CoDiffuseGesture, a diffusion-model framework that reasons about semantic speech content, dialog intention, and context-emotional cues for both participants. It leverages sentence-level representations from DistilBERT for intention, Roberta for emotion, and BERT for contextual semantics, conditioning gesture generation on multimodal inputs as $c=[A, T, I, E]$ and extending diffusion models to dyadic interactions. The approach achieves improved human-likeness and appropriateness over baselines on the GENEA2023 dataset, demonstrating that modeling both participants' intents and emotions yields more natural, diverse gestures. The work has practical implications for lifelike virtual humans and interactive agents, with future work exploring richer social knowledge, environment factors, and named-entity semantics to further enhance realism.

Abstract

Audio-driven co-speech human gesture generation has made remarkable advancements recently. However, most previous works only focus on single person audio-driven gesture generation. We aim at solving the problem of conversational co-speech gesture generation that considers multiple participants in a conversation, which is a novel and challenging task due to the difficulty of simultaneously incorporating semantic information and other relevant features from both the primary speaker and the interlocutor. To this end, we propose CoDiffuseGesture, a diffusion model-based approach for speech-driven interaction gesture generation via modeling bilateral conversational intention, emotion, and semantic context. Our method synthesizes appropriate interactive, speech-matched, high-quality gestures for conversational motions through the intention perception module and emotion reasoning module at the sentence level by a pretrained language model. Experimental results demonstrate the promising performance of the proposed method.

Figures (2)

• Figure 1: (Left) Forward process of CoDiffuseGesture is to add noise to the motion sequence from $t$ = 0 until $t$ = $T$. (Right) Reverse process of CoDiffuseGesture is to learn the denoising ability. A step $t$ and a noisy gesture sequence $X_t$ at the noising step conditioning on $c$ incorporating intent and emotion features $E, I$, audio $A$, seed gesture sequence $d$ are fed to the network.
• Figure 2: Visualization of different gestures reflected by the four variational models.