Learning Human Motion with Temporally Conditional Mamba

TL;DR

This work tackles learning human motion conditioned on time-varying inputs by embedding temporal conditioning directly into the Mamba backbone within a diffusion framework. The Temporally Conditional Mamba (TCM) integrates conditioning signals into the recurrent dynamics, coupled with a Spatial Mamba and Adaptive Layer Norm to preserve temporal and spatial coherence. Empirical results across music-driven dance, egocentric video-based estimation, and object-trajectory tasks show that TCM improves temporal alignment, motion realism, and condition consistency over cross-attention and vanilla Mamba baselines. The approach demonstrates strong generalization across synthesis and estimation tasks and highlights the practical potential for temporally conditioned motion generation in animation, AR/VR, and human-robot interaction.

Abstract

Learning human motion based on a time-dependent input signal presents a challenging yet impactful task with various applications. The goal of this task is to generate or estimate human movement that consistently reflects the temporal patterns of conditioning inputs. Existing methods typically rely on cross-attention mechanisms to fuse the condition with motion. However, this approach primarily captures global interactions and struggles to maintain step-by-step temporal alignment. To address this limitation, we introduce Temporally Conditional Mamba, a new mamba-based model for human motion generation. Our approach integrates conditional information into the recurrent dynamics of the Mamba block, enabling better temporally aligned motion. To validate the effectiveness of our method, we evaluate it on a variety of human motion tasks. Extensive experiments demonstrate that our model significantly improves temporal alignment, motion realism, and condition consistency over state-of-the-art approaches. Our project page is available at https://zquang2202.github.io/TCM.

Figures (6)

• Figure 1: High-level comparison between our approach and previous methods. (a) Previous works usually use Cross-Attention to integrate input condition into Mamba/Transformer backbone. (b) Our approach embeds the condition directly within the Mamba block; (c) We show the head trajectory over time in an ego-to-motion task. Compared to Cross-Attention and Vanilla Mamba, which generate motions that deviate noticeably from the ground truth, our method produces a trajectory that closely follows the actual motion pattern.
• Figure 2: Architecture overview of the proposed approach. (a) We show the overview of the diffusion human motion framework with Mamba blocks. (b) Our key contribution is the Temporally Conditional Mamba, which incorporates temporal conditions into the internal dynamics of the Mamba block. (c) The Spatial Mamba block is used to learn human spatial features.
• Figure 3: Motion and music beat alignment. We plot the mean joint velocity over time for different methods. Kinematic beats are identified as local minima in the velocity curves. Our method produces motion with kinematic beats that align more closely with the music beats.
• Figure 4: t-SNE visualization of $\gamma_{\mathbf{B},\mathbf{C}}$ and $\beta_{\mathbf{B},\mathbf{C}}$ in TCM blocks. Clustering in the t-SNE plot reveals that the learned $\gamma_{\mathbf{B},\mathbf{C}}$ and $\beta_{\mathbf{B},\mathbf{C}}$ parameters encode discriminative information that reflects distinct music genres.
• Figure 5: Qualitative comparison of human motion estimation from the egocentric task. Our method produces more coherent and accurate motion compared to EgoEgo. For additional visualizations, please refer to our demo video.