ExGes: Expressive Human Motion Retrieval and Modulation for Audio-Driven Gesture Synthesis

TL;DR

The paper tackles expressive, semantically aligned audio-driven gesture synthesis by introducing ExGes, a retrieval-enhanced diffusion framework. ExGes combines a Motion Base Construction, a Motion Retrieval Module using contrastive learning and momentum distillation, and a Precise Control Module with partial and stochastic masking to provide fine-grained control. On BEAT2, ExGes achieves lower Fréchet Gesture Distance and higher diversity than strong baselines, with user studies showing substantial preferences for naturalness and semantic relevance. This approach reduces reliance on large auxiliary datasets and enhances expressiveness and alignment between speech and gestures, offering practical improvements for animated avatars and HCI systems.

Abstract

Audio-driven human gesture synthesis is a crucial task with broad applications in virtual avatars, human-computer interaction, and creative content generation. Despite notable progress, existing methods often produce gestures that are coarse, lack expressiveness, and fail to fully align with audio semantics. To address these challenges, we propose ExGes, a novel retrieval-enhanced diffusion framework with three key designs: (1) a Motion Base Construction, which builds a gesture library using training dataset; (2) a Motion Retrieval Module, employing constrative learning and momentum distillation for fine-grained reference poses retreiving; and (3) a Precision Control Module, integrating partial masking and stochastic masking to enable flexible and fine-grained control. Experimental evaluations on BEAT2 demonstrate that ExGes reduces Fréchet Gesture Distance by 6.2\% and improves motion diversity by 5.3\% over EMAGE, with user studies revealing a 71.3\% preference for its naturalness and semantic relevance. Code will be released upon acceptance.

Figures (6)

• Figure 1: Difference between ExGes and existing methods. ExGes aims at generating gestures that are more expressive and semantic aligning with the audio.
• Figure 2: Overview our method. The audio is first used to retrieval to get expressive gestures. The the retrieval frames are served as the control signs that insert into the Precise Control Module, enhancing the expressiveness of generated gestures.
• Figure 3: Different mask type in our work. The blue squares indicates the control information and the white information indicates the noise location.
• Figure 4: Comparison results between EX-Ges with existing methods. We compare the outputs generated by different models under the same speech input and plotted the hand motion sequences with the richest semantic information.
• Figure 5: Visualization results of of the searched keyframes.