Dyadic Mamba: Long-term Dyadic Human Motion Synthesis

TL;DR

This work tackles the generation of realistic dyadic human motion from text descriptions over long time horizons, addressing transformer limitations in long-sequence reasoning due to positional encodings. It introduces Dyadic Mamba, a diffusion-based State-Space Model with per-person self dynamics and a concatenation-based cross-flow that conditions on text via Adaptive LayerNorm, enabling arbitrary-length generation without cross-attention. The method achieves competitive short-term results and significantly improves long-term generation quality compared to transformer-based baselines, and it introduces a new long-term motion quality benchmark to evaluate sustained realism. Overall, the study highlights state-space models as a promising, parameter-efficient direction for long-horizon dyadic motion synthesis from textual descriptions, with practical implications for entertainment and digital content creation.

Abstract

Generating realistic dyadic human motion from text descriptions presents significant challenges, particularly for extended interactions that exceed typical training sequence lengths. While recent transformer-based approaches have shown promising results for short-term dyadic motion synthesis, they struggle with longer sequences due to inherent limitations in positional encoding schemes. In this paper, we introduce Dyadic Mamba, a novel approach that leverages State-Space Models (SSMs) to generate high-quality dyadic human motion of arbitrary length. Our method employs a simple yet effective architecture that facilitates information flow between individual motion sequences through concatenation, eliminating the need for complex cross-attention mechanisms. We demonstrate that Dyadic Mamba achieves competitive performance on standard short-term benchmarks while significantly outperforming transformer-based approaches on longer sequences. Additionally, we propose a new benchmark for evaluating long-term motion synthesis quality, providing a standardized framework for future research. Our results demonstrate that SSM-based architectures offer a promising direction for addressing the challenging task of long-term dyadic human motion synthesis from text descriptions.

Figures (7)

• Figure 1: Given a textual description of a dyadic interaction, our model produces the dyadic human motion for the two persons. Our Mamba-based approach is capable of producing longer sequences than state-of-the-art Transformer-based approaches. The first two rows depict generations from our model, while the last row shows a generation from InterGen liang2024intergen. Our model successfully generates motion sequences exceeding the training length of 10 seconds. Notably, it maintains close contact between the two individuals over extended time horizons. In contrast, transformer-based approaches like InterGen liang2024intergen (last row) struggle with temporal extrapolation, resulting in motion artifacts or even complete breakdown (easier to observe in supplementary video).
• Figure 2: Model Overview: The Dyadic Mamba performs single-step denoising of a $t$-noised dyadic signal $\{\mathbf{x}_a^{(t)}, \mathbf{x}_b^{(t)} \}$ to produce the denoised dyadic signal $\{\mathbf{\hat{x}}_a^{(t-1)}, \mathbf{\hat{x}}_b^{(t-1)} \}$. Our architecture comprises $N$ cooperative blocks that process signals iteratively, culminating in the final motion prediction. Each block integrates text embedding $\mathbf{c}$ and diffusion step embedding $t$ through projection and summation, yielding a conditioning embedding $\mathrm{emb}$ that modulates the network via Adaptive LayerNorm perez2018filmpeebles2023scalable. The processing pipeline begins with individual motion sequences undergoing linear projection to align their dimensionality before being processed by an individual Mamba module $m_{\mathrm{self}}^{(n)}$ to obtain $\mathbf{\overline{x}}_{\circ}^{(t)}$ where $\circ \in \{a,b\}$. Subsequently, the two individual motion sequences $\mathbf{\overline{x}}_{a}^{(t)}$ and $\mathbf{\overline{x}}_{b}^{(t)}$ are concatenated and linearly projected by $d^{(n)}$ to re-align the dimensionality. The resulting signal contains information about both persons and is passed to the cooperative Mamba block $m_{\mathrm{cross}}^{(n)}$. The final output is obtained through linear projection to the target dimensionality. Notably, parameter sharing across individual motion sequence processing ensures both computational efficiency and order invariance in the dyadic motion representation.
• Figure 3: Qualitative Results: Dyadic motion generation results for various text descriptions.
• Figure 4: Per-frame long-term per-person motion quality (NDMS tanke2021intention$\uparrow$) evaluated on the InterHuman dataset liang2024intergen across multiple temporal horizons (7s, 14s, and 28s). Our approach demonstrates consistent performance compared to InterGenliang2024intergen and its variant InterGen (RoPE)liang2024intergensu2024roformer. While InterGen exhibits degradation when generating sequences beyond its 10s training horizon, manifesting as increasingly apparent motion artifacts, our method maintains realistic motion quality across all evaluated temporal scales. The RoPE-enhanced variant successfully extends InterGen's effective generation window to approximately twice the training sequence length ($\approx$20s), but fails to maintain coherence when generating longer sequences (28s). All three models were trained on sequence length of at most 10s.
• Figure 5: Conditioning: We experiment with two variants for conditioning the Dyadic Mamba on diffusion step $t$ and text embedding $\mathbf{c}$: (a) Prepending approach: The conditioning embeddings are simply prepended to the motion sequence before passing it to the Mamba module. (b) Adaptive LayerNorm modulation: The input signal is scaled and shifted by a linearly projected addition of $t$ and $\mathbf{c}$ before being passed to the Mamba module.