DIDiffGes: Decoupled Semi-Implicit Diffusion Models for Real-time Gesture Generation from Speech

TL;DR

DIDiffGes tackles the bottleneck of slow diffusion-based gesture generation by introducing a Decoupled Semi-Implicit Diffusion framework that enables real-time synthesis of high-quality co-speech gestures. The method combines GAN-assisted large-step sampling with a sequential denoiser that first reconstructs local motion and then conditions global/root motion, while decoupling body and hand noise distributions. A semi-implicit objective couples adversarial learning with explicit reconstruction losses and an Auxiliary Forward Diffusion Constraint to model complex large-step distributions robustly. Experimental results on BEATs, ZeroEGGs, and AIST++ demonstrate state-of-the-art quality and a major speed-up (10 steps, ≈0.4s for 88 frames), highlighting strong practical potential for real-time human–AI interaction and cross-domain gesture generation such as dance. The work offers a scalable path toward real-time, expressive gesture generation driven by speech, with implications for real-time avatars and multimodal communication systems.

Abstract

Diffusion models have demonstrated remarkable synthesis quality and diversity in generating co-speech gestures. However, the computationally intensive sampling steps associated with diffusion models hinder their practicality in real-world applications. Hence, we present DIDiffGes, for a Decoupled Semi-Implicit Diffusion model-based framework, that can synthesize high-quality, expressive gestures from speech using only a few sampling steps. Our approach leverages Generative Adversarial Networks (GANs) to enable large-step sampling for diffusion model. We decouple gesture data into body and hands distributions and further decompose them into marginal and conditional distributions. GANs model the marginal distribution implicitly, while L2 reconstruction loss learns the conditional distributions exciplictly. This strategy enhances GAN training stability and ensures expressiveness of generated full-body gestures. Our framework also learns to denoise root noise conditioned on local body representation, guaranteeing stability and realism. DIDiffGes can generate gestures from speech with just 10 sampling steps, without compromising quality and expressiveness, reducing the number of sampling steps by a factor of 100 compared to existing methods. Our user study reveals that our method outperforms state-of-the-art approaches in human likeness, appropriateness, and style correctness. Project is https://cyk990422.github.io/DIDiffGes.

Figures (2)

• Figure 1: Comparison of four different sampling methods: DSG yang2023diffusestylegesture with DDPM, DSG with DDIM, our method with 1000-step sampling, and our method with 10-step sampling.
• Figure 2: Our learning framework integrates a Sequential Diffusion Denoiser with two transformer encoders and a Decoupled Semi-implicit Objective. The first encoder denoises local motion and provides a conditional signal for the second encoder, which denoises root noise. The final result, a combination of local motion and root result, is added with t-1 step noise via posterior sampling and then decoupled into body and hand noise. These noise undergo adversarial training against prior sampled noise, supervised by Auxiliary Forward Diffusion Loss. For a detailed description of the network architecture, please refer to our supplementary materials.