DiM-Gestor: Co-Speech Gesture Generation with Adaptive Layer Normalization Mamba-2

TL;DR

DiM-Gestor, an innovative end-to-end generative model leveraging the Mamba-2 architecture, which enables precise modeling of the nuanced interplay between speech features and gesture dynamics and releases the CCG dataset, a Chinese Co-Speech Gestures dataset.

Abstract

Speech-driven gesture generation using transformer-based generative models represents a rapidly advancing area within virtual human creation. However, existing models face significant challenges due to their quadratic time and space complexities, limiting scalability and efficiency. To address these limitations, we introduce DiM-Gestor, an innovative end-to-end generative model leveraging the Mamba-2 architecture. DiM-Gestor features a dual-component framework: (1) a fuzzy feature extractor and (2) a speech-to-gesture mapping module, both built on the Mamba-2. The fuzzy feature extractor, integrated with a Chinese Pre-trained Model and Mamba-2, autonomously extracts implicit, continuous speech features. These features are synthesized into a unified latent representation and then processed by the speech-to-gesture mapping module. This module employs an Adaptive Layer Normalization (AdaLN)-enhanced Mamba-2 mechanism to uniformly apply transformations across all sequence tokens. This enables precise modeling of the nuanced interplay between speech features and gesture dynamics. We utilize a diffusion model to train and infer diverse gesture outputs. Extensive subjective and objective evaluations conducted on the newly released Chinese Co-Speech Gestures dataset corroborate the efficacy of our proposed model. Compared with Transformer-based architecture, the assessments reveal that our approach delivers competitive results and significantly reduces memory usage, approximately 2.4 times, and enhances inference speeds by 2 to 4 times. Additionally, we released the CCG dataset, a Chinese Co-Speech Gestures dataset, comprising 15.97 hours (six styles across five scenarios) of 3D full-body skeleton gesture motion performed by professional Chinese TV broadcasters.

Figures (11)

• Figure 1: We propose DiM-Gestor, an end-to-end AdaLN Mamba-2 and diffusion-based architecture for co-speech gesture generation. In addition, we present the comprehensive Chinese Co-Speech Gestures (CCG) dataset, comprising 15.97 hours of full-body gesture motion performed by professional Chinese TV broadcasters. This dataset encompasses six distinct styles across five scenarios.
• Figure 2: The architecture of DiM-Gestor incorporates a Mamba-2 fuzzy feature extractor and an Adaptive Layer Normalization (AdaLN) Mamba-2 diffusion architecture. The fuzzy feature extractor features a dual-component system engineered to capture the nuanced style and detailed audio features into unified latent features. These unified latent features are subsequently channeled into the AdaLN Mamba-2. This module is pivotal in modeling the intricate relationship between the incoming audio features and the corresponding gestures. It facilitates the estimation of diffusion noise within the diffusion model, ensuring the generation of diverse gestures. The overall schematic includes three main components: (a) Mamba-2 Fuzzy Feature Extractor, (b) Stack of AdaLN Mamba-2 Blocks, (c) Gestures Encoder and Decoder, and (d) Denoising Diffusion Probabilistic Model (DDPM).
• Figure 3: The Structured Masked Attention.
• Figure 4: An overview of the recorded Chinese TV broadcasters dataset.
• Figure 5: The female broadcaster is presenting neutral programs live on air (Left: GT; Center: DiM-Gestor; Right: PG-12blocks).