SmoothSync: Dual-Stream Diffusion Transformers for Jitter-Robust Beat-Synchronized Gesture Generation from Quantized Audio

TL;DR

SmoothSync tackles co-speech gesture generation by addressing jitter, foot-sliding, and limited sampling diversity through a dual-stream diffusion transformer that fuses quantized audio tokens with motion tokens. The method introduces a jitter-suppress loss and a quantized Mel-spectrogram input to enable diverse yet synchronized gestures, plus a robust Smooth-BC metric for rhythm evaluation. Key contributions include the dual-stream DiT architecture, probabilistic audio quantization, and a segment-based long-motion generation strategy that preserves temporal coherence. On BEAT2 and SHOW, SmoothSync achieves state-of-the-art motion realism, rhythmic alignment, and diversity, while delivering real-time performance, making it suitable for interactive avatars and embodied AI systems.

Abstract

Co-speech gesture generation is a critical area of research aimed at synthesizing speech-synchronized human-like gestures. Existing methods often suffer from issues such as rhythmic inconsistency, motion jitter, foot sliding and limited multi-sampling diversity. In this paper, we present SmoothSync, a novel framework that leverages quantized audio tokens in a novel dual-stream Diffusion Transformer (DiT) architecture to synthesis holistic gestures and enhance sampling variation. Specifically, we (1) fuse audio-motion features via complementary transformer streams to achieve superior synchronization, (2) introduce a jitter-suppression loss to improve temporal smoothness, (3) implement probabilistic audio quantization to generate distinct gesture sequences from identical inputs. To reliably evaluate beat synchronization under jitter, we introduce Smooth-BC, a robust variant of the beat consistency metric less sensitive to motion noise. Comprehensive experiments on the BEAT2 and SHOW datasets demonstrate SmoothSync's superiority, outperforming state-of-the-art methods by -30.6% FGD, 10.3% Smooth-BC, and 8.4% Diversity on BEAT2, while reducing jitter and foot sliding by -62.9% and -17.1% respectively. The code will be released to facilitate future research.

Figures (6)

• Figure 1: SmoothSync uses quantized audio mel-energy as input and efficiently fuses audio and motion features through a dual-stream network for denoising generation. Our method produces smoother motions with reduced foot sliding and global movement ranges that better match real data. In contrast, previous methods generate motions with severe drift and excessive jittering. Zoom in for better view.
• Figure 2: Overview of our Dual-Stream DiT architecture. The framework processes audio and motion through separate pathways before fusing them via specialized transformer blocks, enabling high-quality speech-driven motion generation.
• Figure 3: Smooth-BC metric is proposed to exclude motion beats falsely detected due to jitter artifacts by imposing strict constraints on the slope around velocity extrema.
• Figure 4: Comparison on BEAT2 Dataset. Compared to baseline methods, our method generates motion that appropriately pauses during speech pauses, performs correct up-and-down hand movements at emphasis points following speech rhythm, and produces gestures closer to ground truth with semantically appropriate hand movements at strong semantic cues like "You should". At "how loud", SmoothSync generates large-amplitude outward stretching movements with vertical directions that don't completely match ground truth, demonstrating SmoothSync's diversity capability.
• Figure 5: Comparison on SHOW Dataset. Compared to TalkSHOW's limited motion range, our method generates gestures with larger amplitudes and greater diversity, exhibiting rhythmic patterns that align more closely with ground truth. SmoothSync also demonstrates superior semantic matching: at emphasized words such as "just" and "they", our approach produces gestures that better correspond to the semantic content and prosodic stress of speech.