Conditional GAN for Enhancing Diffusion Models in Efficient and Authentic Global Gesture Generation from Audios

TL;DR

A conditional GAN is introduced to capture audio control signals and implicitly match the multimodal denoising distribution between the diffusion and denoising steps within the same sampling step, aiming to sample larger noise values and apply fewer denoising steps for high-speed generation.

Abstract

Audio-driven simultaneous gesture generation is vital for human-computer communication, AI games, and film production. While previous research has shown promise, there are still limitations. Methods based on VAEs are accompanied by issues of local jitter and global instability, whereas methods based on diffusion models are hampered by low generation efficiency. This is because the denoising process of DDPM in the latter relies on the assumption that the noise added at each step is sampled from a unimodal distribution, and the noise values are small. DDIM borrows the idea from the Euler method for solving differential equations, disrupts the Markov chain process, and increases the noise step size to reduce the number of denoising steps, thereby accelerating generation. However, simply increasing the step size during the step-by-step denoising process causes the results to gradually deviate from the original data distribution, leading to a significant drop in the quality of the generated actions and the emergence of unnatural artifacts. In this paper, we break the assumptions of DDPM and achieves breakthrough progress in denoising speed and fidelity. Specifically, we introduce a conditional GAN to capture audio control signals and implicitly match the multimodal denoising distribution between the diffusion and denoising steps within the same sampling step, aiming to sample larger noise values and apply fewer denoising steps for high-speed generation.

Figures (4)

• Figure 1: Our method generates global gesture motion highly matched with input audio melody in a short runtime. On the (a) BEAT dataset and (b) ZeroEGGs dataset, our method's average runtime is 5ms and 4.6ms per frame, respectively. For reference, DSG's corresponding times are 5.2 and 4.8s. (c) The overall comparison of inference time cost and FGD on both datasets. In the figure, we compare the per-frame runtime with the state-of-the-art methods' FGD.
• Figure 2: Network architecture. During training, we introduce a GAN structure based on conditional denoising diffusion to capture the complex distribution of gesture sequences in a multi-step process, enabling larger sampling step sizes. During inference, we use large step sizes and fewer steps for sampling, according to the input audio control signal, to achieve fast, high-quality gesture sequences, thus supporting real-time tasks.
• Figure 3: Conditional GAN architecture. The Conditional Gesture Generator is employed for the audio-to-gesture task, where z is the latent variable sampled from a Gaussian distribution. It consists of a Transformer Encoder, Linear, and Mask components. The Conditional Gesture Discriminator is utilized to distinguish between genuine and counterfeit gesture sequences, comprising Linear, SELU, and Group Norm modules.
• Figure 4: In comparison to contemporary diffusion-based methods such as DSG and non-diffusion methods like CAMN, our approach achieves high-speed generation while ensuring the utmost generation quality.