Mitigating Error Accumulation in Co-Speech Motion Generation via Global Rotation Diffusion and Multi-Level Constraints

TL;DR

The paper tackles hierarchical error accumulation in co-speech motion generation by switching from local to global joint rotations, predicting $R_k^{global}$ directly to avoid recursive FK drift. GlobalDiff uses a diffusion-based framework conditioned on seed pose and prosody, and introduces a three-tier constraint scheme: joint-level virtual anchors, skeleton-level Angular Matrix for inter-bone relations, and a motion-level multi-scale variational encoder for temporal coherence. The approach achieves state-of-the-art results on BEAT2 across multiple speakers, demonstrating improved stability, anatomical plausibility, and expressive co-speech gestures with a reported ~46% performance gain over previous SOTA. This work significantly enhances robustness and generalization in holistic co-speech motion generation and provides a concrete blueprint for structurally-aware global-rotation diffusion methods.

Abstract

Reliable co-speech motion generation requires precise motion representation and consistent structural priors across all joints. Existing generative methods typically operate on local joint rotations, which are defined hierarchically based on the skeleton structure. This leads to cumulative errors during generation, manifesting as unstable and implausible motions at end-effectors. In this work, we propose GlobalDiff, a diffusion-based framework that operates directly in the space of global joint rotations for the first time, fundamentally decoupling each joint's prediction from upstream dependencies and alleviating hierarchical error accumulation. To compensate for the absence of structural priors in global rotation space, we introduce a multi-level constraint scheme. Specifically, a joint structure constraint introduces virtual anchor points around each joint to better capture fine-grained orientation. A skeleton structure constraint enforces angular consistency across bones to maintain structural integrity. A temporal structure constraint utilizes a multi-scale variational encoder to align the generated motion with ground-truth temporal patterns. These constraints jointly regularize the global diffusion process and reinforce structural awareness. Extensive evaluations on standard co-speech benchmarks show that GlobalDiff generates smooth and accurate motions, improving the performance by 46.0 % compared to the current SOTA under multiple speaker identities.

Figures (7)

• Figure 1: Overview of our motivation and solution. Local rotation diffusion leads to error accumulation in distal joints. Global rotation diffusion avoids this but lacks structural priors (top left). We address this with constraints at the joint, skeleton, and motion levels to ensure coherent and reasonable motion (bottom).
• Figure 2: Overview of the GlobalDiff Framework. Our model generates consistent and expressive co-speech motion using the global rotation diffusion augmented with multi-level structural constraints. The diffusion model is conditioned on seed pose and prosodic features and predicts body motion through stacked motion generation blocks. To enforce structural plausibility, we introduce: (a) a Joint structure constraint using virtual anchor points to disambiguate orientations; (b) a Skeleton structure constraint that enforces angular consistency across adjacent bones by aligning the angular matrices; and (c) a Temporal structure constraint based on a shared multi-scale VAE encoder to preserve temporal dynamics. Facial expressions are generated in parallel from prosody using a transformer encoder.
• Figure 3: Visual comparison. Our GlobalDiff produces semantically meaningful gestures that align closely with the spoken content and speaker identity. For example, for the phrase “stop,” Our GlobalDiff generates symmetric and contextually appropriate hand gestures near the head, conveying a clear intent. In contrast, HoloGest and EMAGE often result in unbalanced or semantically incoherent motions, such as asymmetric arms or ambiguous limb orientations.
• Figure 4: Qualitative study on the effect of $\mathcal{L}_j$. Without $\mathcal{L}_j$, finger poses often appear anatomically implausible.
• Figure 5: Qualitative study on the effect of $\mathcal{L}_s$. Without $\mathcal{L}_s$, motion becomes structurally incoherent, exhibiting unbalanced posture and unstable foot placement.