Towards Variable and Coordinated Holistic Co-Speech Motion Generation

TL;DR

This work tackles the challenge of generating realistic and diverse holistic co-speech motion for 3D avatars by endowing a unified probabilistic framework, ProbTalk, with a multi-component pipeline. It introduces PQ-VAE to obtain a rich, discrete representation of holistic motion, a non-autoregressive MaskGIT-inspired predictor augmented with 2D positional encoding to efficiently sample PQ codes, and a refinement stage to sharpen high-frequency details, all under multi-modal conditioning including motion context and speaker identity. The method jointly models facial expressions, hand gestures, and body poses to achieve coordinated, natural movements synchronized with speech, outperforming state-of-the-art approaches on the SHOW dataset in realism and diversity while maintaining competitive inference speed. The approach enables controllable, mode-rich co-speech motion generation with potential applications in immersive avatars and human-robot interaction, and the authors release code and models for research use.

Abstract

This paper addresses the problem of generating lifelike holistic co-speech motions for 3D avatars, focusing on two key aspects: variability and coordination. Variability allows the avatar to exhibit a wide range of motions even with similar speech content, while coordination ensures a harmonious alignment among facial expressions, hand gestures, and body poses. We aim to achieve both with ProbTalk, a unified probabilistic framework designed to jointly model facial, hand, and body movements in speech. ProbTalk builds on the variational autoencoder (VAE) architecture and incorporates three core designs. First, we introduce product quantization (PQ) to the VAE, which enriches the representation of complex holistic motion. Second, we devise a novel non-autoregressive model that embeds 2D positional encoding into the product-quantized representation, thereby preserving essential structure information of the PQ codes. Last, we employ a secondary stage to refine the preliminary prediction, further sharpening the high-frequency details. Coupling these three designs enables ProbTalk to generate natural and diverse holistic co-speech motions, outperforming several state-of-the-art methods in qualitative and quantitative evaluations, particularly in terms of realism. Our code and model will be released for research purposes at https://feifeifeiliu.github.io/probtalk/.

Figures (6)

• Figure 1: Holistic co-speech motion generation examples. Given a speech signal as input, our approach generates variable and coordinated holistic body motions. From top to bottom: the speech transcript, the corresponding audio, and three generated samples. In particular, to emphasize important keywords, our method ensures that facial expressions, head movements, and body motions work in unison.
• Figure 2: Overview of the proposed ProbTalk, a unified probabilistic framework designed to jointly model facial, hand, and body movements in speech. Specifically, ProbTalk first learns a PQ-VAE of holistic body motion, where the latent space is partitioned into subspaces, each of which is quantized using a dedicated codebook. Then, we predict the PQ codes based on the masked code context. In each iteration, we add 2D positional encoding (2D-PE) to embeddings of masked PQ codes to conserve the original structural integrity of the PQ codes. The predicted motion is refined in the secondary stage, further sharpening the high-frequency details.
• Figure 3: Our framework is designed to generate co-speech motion, utilizing multi-modal conditions. In detail, the audio and motion context are individually processed by their respective condition encoders. Following this, the encoded outputs are concatenated and forwarded to a cross-attention layer. Besides, an AdaIN layer huang2017arbitrary is integrated to facilitate the incorporation of speaker identity.
• Figure 4: Qualitative comparison with SOTA methods. The co-speech motion generated by ProbTalk is more realistic, especially in terms of the timing, magnitude, and frequency of movements. We highlight the arm movements in grey. Best viewed in color.
• Figure 5: User study.