Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M3G: Multi-Granular Gesture Generator for Audio-Driven Full-Body Human Motion Synthesis

Zhizhuo Yin, Yuk Hang Tsui, Pan Hui

TL;DR

M3G tackles the challenge of audio-driven holistic gesture synthesis by introducing multi-granular motion modeling. It combines MG-VQ-VAE, which tokenizes motion at multiple temporal scales into a shared codebook, with a Multi-Granular Token Predictor that maps audio and transcripts to these tokens across face, upper body, hands, and lower body, enabling coherent reconstruction via TransTCN decoders. The approach yields stronger objective metrics and favorable perceptual results compared to state-of-the-art methods, demonstrating improved realism, expressiveness, and synchronization. This framework advances virtual avatar realism for metaverse applications by capturing diverse gesture patterns across all body parts with variable granularity.

Abstract

Generating full-body human gestures encompassing face, body, hands, and global movements from audio is a valuable yet challenging task in virtual avatar creation. Previous systems focused on tokenizing the human gestures framewisely and predicting the tokens of each frame from the input audio. However, one observation is that the number of frames required for a complete expressive human gesture, defined as granularity, varies among different human gesture patterns. Existing systems fail to model these gesture patterns due to the fixed granularity of their gesture tokens. To solve this problem, we propose a novel framework named Multi-Granular Gesture Generator (M3G) for audio-driven holistic gesture generation. In M3G, we propose a novel Multi-Granular VQ-VAE (MGVQ-VAE) to tokenize motion patterns and reconstruct motion sequences from different temporal granularities. Subsequently, we proposed a multi-granular token predictor that extracts multi-granular information from audio and predicts the corresponding motion tokens. Then M3G reconstructs the human gestures from the predicted tokens using the MGVQ-VAE. Both objective and subjective experiments demonstrate that our proposed M3G framework outperforms the state-of-the-art methods in terms of generating natural and expressive full-body human gestures.

M3G: Multi-Granular Gesture Generator for Audio-Driven Full-Body Human Motion Synthesis

TL;DR

M3G tackles the challenge of audio-driven holistic gesture synthesis by introducing multi-granular motion modeling. It combines MG-VQ-VAE, which tokenizes motion at multiple temporal scales into a shared codebook, with a Multi-Granular Token Predictor that maps audio and transcripts to these tokens across face, upper body, hands, and lower body, enabling coherent reconstruction via TransTCN decoders. The approach yields stronger objective metrics and favorable perceptual results compared to state-of-the-art methods, demonstrating improved realism, expressiveness, and synchronization. This framework advances virtual avatar realism for metaverse applications by capturing diverse gesture patterns across all body parts with variable granularity.

Abstract

Generating full-body human gestures encompassing face, body, hands, and global movements from audio is a valuable yet challenging task in virtual avatar creation. Previous systems focused on tokenizing the human gestures framewisely and predicting the tokens of each frame from the input audio. However, one observation is that the number of frames required for a complete expressive human gesture, defined as granularity, varies among different human gesture patterns. Existing systems fail to model these gesture patterns due to the fixed granularity of their gesture tokens. To solve this problem, we propose a novel framework named Multi-Granular Gesture Generator (M3G) for audio-driven holistic gesture generation. In M3G, we propose a novel Multi-Granular VQ-VAE (MGVQ-VAE) to tokenize motion patterns and reconstruct motion sequences from different temporal granularities. Subsequently, we proposed a multi-granular token predictor that extracts multi-granular information from audio and predicts the corresponding motion tokens. Then M3G reconstructs the human gestures from the predicted tokens using the MGVQ-VAE. Both objective and subjective experiments demonstrate that our proposed M3G framework outperforms the state-of-the-art methods in terms of generating natural and expressive full-body human gestures.
Paper Structure (22 sections, 17 equations, 4 figures, 4 tables)

This paper contains 22 sections, 17 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Human 3D Motion Generation Approaches
  4. Audio-driven Full-body Gesture Generation
  5. Methodology
  6. Problem Formulation
  7. Multi-Granular Gesture Generator (M3G)
  8. Multi-Granular VQ-VAE (MGVQ-VAE)
  9. Multi-Granular Token Predictor
  10. Audio and Context Feature Fusion
  11. Latent Generation and Token Prediction
  12. Experiments
  13. Datasets
  14. Evaluation Metrics
  15. Comparison Methods
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Illustration of gesture sequences with expressive motion patterns in different granularities.
  • Figure 2: The Overall Workflow of M3G. We illustrate the workflow in two stages: 1. Training MGVQ-VAE to learn discrete latent representations of motions, as known as tokens, for encode and reconstruction. 2. Training Multi-Granular Token Predictor to predict the discrete latent representations from audio.
  • Figure 3: Perceptual study results on motion generation.
  • Figure 4: Qualitative results.