ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis

TL;DR

ConvoFusion is presented, a diffusion-based approach for multi-modal gesture synthesis, which can not only generate gestures based on multi-modal speech inputs, but can also facilitate controllability in gesture synthesis.

Abstract

Gestures play a key role in human communication. Recent methods for co-speech gesture generation, while managing to generate beat-aligned motions, struggle generating gestures that are semantically aligned with the utterance. Compared to beat gestures that align naturally to the audio signal, semantically coherent gestures require modeling the complex interactions between the language and human motion, and can be controlled by focusing on certain words. Therefore, we present ConvoFusion, a diffusion-based approach for multi-modal gesture synthesis, which can not only generate gestures based on multi-modal speech inputs, but can also facilitate controllability in gesture synthesis. Our method proposes two guidance objectives that allow the users to modulate the impact of different conditioning modalities (e.g. audio vs text) as well as to choose certain words to be emphasized during gesturing. Our method is versatile in that it can be trained either for generating monologue gestures or even the conversational gestures. To further advance the research on multi-party interactive gestures, the DnD Group Gesture dataset is released, which contains 6 hours of gesture data showing 5 people interacting with one another. We compare our method with several recent works and demonstrate effectiveness of our method on a variety of tasks. We urge the reader to watch our supplementary video at our website.

Figures (11)

• Figure 1: Our ConvoFusion approach generates body and hand gestures in monadic and dyadic settings, while also offering advanced control over textual and auditory modalities in speech.Lastly, we introduce the DnD Group Gesture dataset, showcasing rich interactions with co-speech gestures between five participants. Motions rendered using ASH zhu2023ash.
• Figure 2: Overview of the proposed approach. We generate gestures conditioned on multiple conditioning signals such as text, audio, speaker style, etc. using a latent diffusion approach. During inference, we introduce modality guidance and word-excitation guidance to control the properties of the generated gestures.
• Figure 3: The model schema. Given a training motion $\mathbf{x} \in \mathbb{R}^{N\times J \times 3}$, we first extract its latent encoding $\mathbf{\hat{z}}^{(0)}$ (\ref{['ssec-m:uncond']}), which is then denoised by a network that incorporates the various modalities in the denoising process. At inference time, the denoised latents are decoded to produce the final generation, $\mathbf{\hat{x}}$ (\ref{['ssec-m:cond-gesture-gen']}). During this process, our method allows to control the generation through coarse-grained modality guidance or fine-grained word-excitation guidance (\ref{['ssec-m:guidance-control']}). Dotted lines represent components used only during inference.
• Figure 4: Chunked latent encoding-decoding. We encode a motion of $N$ frames into a sequence of $M$ latent vectors, which are jointly decoded by the decoder $\mathcal{D}$. Encoding into chunked latents allows for perpetual rollout and decoding jointly induces temporal consistency while converting the latents back into motion.
• Figure 5: Results of the user study. We compare with CaMN liu2022beat and MLD mld, and achieve an overall favourable preference scores for monadic and dyadic settings. We also evaluate the effectiveness of the word-excitation guidance (WEG).