MimicParts: Part-aware Style Injection for Speech-Driven 3D Motion Generation

TL;DR

MimicParts tackles the challenge of generating stylized 3D motion from speech by introducing part-aware style injection and a part-aware diffusion model. By partitioning the body into regions and learning localized style representations, the framework captures fine-grained regional differences and dynamically modulates motion according to rhythm and emotion through region-specific attention. The approach leverages SMPL-X, RVQ-VAE latent spaces, and a two-stage training regime with contrastive style learning, achieving superior style fidelity, motion-speech alignment, and perceptual naturalness on BEAT2, supported by extensive quantitative and user studies. This work advances realistic, expressive, and regionally coherent speech-driven motion generation with practical implications for VR, animation, and embodied AI.

Abstract

Generating stylized 3D human motion from speech signals presents substantial challenges, primarily due to the intricate and fine-grained relationships among speech signals, individual styles, and the corresponding body movements. Current style encoding approaches either oversimplify stylistic diversity or ignore regional motion style differences (e.g., upper vs. lower body), limiting motion realism. Additionally, motion style should dynamically adapt to changes in speech rhythm and emotion, but existing methods often overlook this. To address these issues, we propose MimicParts, a novel framework designed to enhance stylized motion generation based on part-aware style injection and part-aware denoising network. It divides the body into different regions to encode localized motion styles, enabling the model to capture fine-grained regional differences. Furthermore, our part-aware attention block allows rhythm and emotion cues to guide each body region precisely, ensuring that the generated motion aligns with variations in speech rhythm and emotional state. Experimental results show that our method outperforming existing methods showcasing naturalness and expressive 3D human motion sequences.

Figures (5)

• Figure 1: Speech-driven stylized 3D motion generation aims to capture individual style differences and generate stylized motions synchronized with speech. Prior works often employ a global style extractor (a) that derives a single full-body style embedding from an motion sequence during training. This approach tends to capture only the overall motion style, overlooking regional style differences, which leads to reduced diversity and accuracy in generation. In contrast, our part-aware style extractor (b) focuses on learning finer-grained representations, enabling better capture and expression of regional style differences.
• Figure 2: Overview of the MimicParts framework, which consists of a part-aware style encoder, a multi-level speech encoder, and a part-aware denoising module. Part-aware Style Encoder: Given a reference motion and body-part masks, localized style representations for the upper body, lower body, and hands are extracted, enabling part-aware style injection. Part-aware Denoising Process: Guided by speech features and localized style representations, the part-aware diffusion model progressively denoises $z_T$. A part‑aware self attention block fuses concent features and local motion representations. Rhythm features and emotion features are then integrated via cross‐attention to guide each body region precisely. R-VQ Decoder: The final latent is quantized via body-part-specific codebooks and decoded into stylized 3D motion, achieving both full-body coherence and body-part-level stylistic diversity.
• Figure 3: Architecture of Part-Aware Attention Block. It allocates different attention heads to specific body regions, ensuring regional modeling, where $f_u$, $f_l$, and $f_h$ represent features of the upper body, lower body, and hands, respectively. During cross-attention, rhythm ($a_r$) or emotion ($a_e$) features are used as the keys and values.
• Figure 4: Qualitative comparison on the BEAT2 dataset. The subject demonstrates limited lower-body movement and subtle upper-body gestures. Our method generates highly synchronized full-body motion that closely matches the subject's natural style. Blue arrows indicate the trend of hand movements, while red arrows represent the trend of leg movements.
• Figure 5: User Study. Our MimicParts consistently receives higher user ratings across human-likeness, speech-gesture appropriateness, and style-accuracy, with a clear margin over all baselines.