3DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Control

TL;DR

3DGesPolicy reframes holistic co-speech gesture generation as a trajectory-control problem learned via a diffusion policy, enabling stable, semantically coherent full-body and facial motion conditioned on speech and phonemes. A dedicated GAP fusion module aligns audio, phoneme, and gesture signals at phoneme level, yielding precise lip-sync and expressive gestures. Extensive BEAT2 evaluations show significant improvements over state-of-the-art methods in realism, synchronization, and expressiveness, with strong qualitative and user-study support. This approach bridges perception and control for 3D digital humans, promising more natural and communicative virtual agents in AR/VR and robotics contexts, and points to future work on finer hand and finger articulation with holistic coordination.

Abstract

Generating holistic co-speech gestures that integrate full-body motion with facial expressions suffers from semantically incoherent coordination on body motion and spatially unstable meaningless movements due to existing part-decomposed or frame-level regression methods, We introduce 3DGesPolicy, a novel action-based framework that reformulates holistic gesture generation as a continuous trajectory control problem through diffusion policy from robotics. By modeling frame-to-frame variations as unified holistic actions, our method effectively learns inter-frame holistic gesture motion patterns and ensures both spatially and semantically coherent movement trajectories that adhere to realistic motion manifolds. To further bridge the gap in expressive alignment, we propose a Gesture-Audio-Phoneme (GAP) fusion module that can deeply integrate and refine multi-modal signals, ensuring structured and fine-grained alignment between speech semantics, body motion, and facial expressions. Extensive quantitative and qualitative experiments on the BEAT2 dataset demonstrate the effectiveness of our 3DGesPolicy across other state-of-the-art methods in generating natural, expressive, and highly speech-aligned holistic gestures.

Figures (5)

• Figure 1: Comparison of holistic co-speech gesture generation paradigms. Unlike part-based or frame-level methods that use discrete codebooks or absolute gesture regression, 3DGesPolicy reformulates the task as an action-based trajectory control problem. By refining unified gesture actions in phoneme level through a diffusion policy training loop, our framework ensures stable, spatially and semantically coherent movements.
• Figure 2: Overview of 3DGesPolicy architecture. Our architecture first disentangles history gesture into action sequences, then Perception module encodes gesture, audio and phoneme sequences into observation representations that serve as conditions for Decision module with GAP fusion module, where actions are produced through denoising and drive the template to output following holistic gestures.
• Figure 3: Qualitative comparison of holistic gesture generation on the BEAT2 dataset. Our method produces more expressive and speech-aligned body movements while avoiding static or semantically irrelevant gestures, and achieves superior lip-speech synchronization with accurate capture of subtle phonetic details compared to baseline methods.
• Figure 4: Additional experiment results of gesture generation and facial generation on BEAT2 datasets.
• Figure 5: Designed user study interface. Each participant need to answer 12 video pairs and here only one video pair is shown due to the page limit.