Multi-granular body modeling with Redundancy-Free Spatiotemporal Fusion for Text-Driven Motion Generation

TL;DR

HiSTF Mamba is introduced, a framework with three parts: Dual-Spatial Mamba, Bi-Temporal Mamba and a Dynamic Spatiotemporal Fusion Module (DSFM), which removes redundant temporal information, extracts complementary cues and fuses them with spatial features to build a richer spatiotemporal representation.

Abstract

Text-to-motion generation sits at the intersection of multimodal learning and computer graphics and is gaining momentum because it can simplify content creation for games, animation, robotics and virtual reality. Most current methods stack spatial and temporal features in a straightforward way, which adds redundancy and still misses subtle joint-level cues. We introduce HiSTF Mamba, a framework with three parts: Dual-Spatial Mamba, Bi-Temporal Mamba and a Dynamic Spatiotemporal Fusion Module (DSFM). The Dual-Spatial module runs part-based and whole-body models in parallel, capturing both overall coordination and fine-grained joint motion. The Bi-Temporal module scans sequences forward and backward to encode short-term details and long-term dependencies. DSFM removes redundant temporal information, extracts complementary cues and fuses them with spatial features to build a richer spatiotemporal representation. Experiments on the HumanML3D benchmark show that HiSTF Mamba performs well across several metrics, achieving high fidelity and tight semantic alignment between text and motion.

Figures (5)

• Figure 1: Qualitative visualization of HiSTF Mamba's generated motions.
• Figure 2: The overview of the proposed HiSTF Mamba.(a) Sampling Process. The model receives a text prompt as input and begins the iterative generation process from Gaussian noise, ultimately producing a noise-free action sequence $x_0$ that is semantically aligned with the prompt. (b) HiSTF Mamba Module. The HiSTF Mamba module primarily aims to predict a clean motion sequence $\{\hat{{x}}_{0}^{1},\hat{{x}}_{0}^{2},\dots,\hat{{x}}_{0}^{N}\}$ from a noisy input sequence $\{x_T^1, x_T^2, \ldots, x_T^N\}$. By leveraging Bi-Temporal Mamba and Dual-Spatial Mamba, the model extracts temporal features $\{T_{1}, \dots, T_{N}\}$ and a spatial feature $S$. These $N$ temporal features are then fed into the DSFM module to remove redundancy and are subsequently fused with $S$, producing a spatiotemporal motion representation that captures both joint-level details and overall body coordination. About Whole2Parts and Parts2Whole, see the appendix \ref{['A4']}
• Figure 3: Qualitative comparison of typical methods. We randomly selected three textual samples to generate motion sequences. Our results exhibit closer alignment with the text and present richer joint-level details.
• Figure 4: Comparison of HiSTF Mamba’s AITS and FID with other baseline methods.
• Figure 5: Illustration of the part‐based and whole‐based skeletal scans on the HumanML3D guo2022generating (left) and KIT plappert2016kit (right) datasets.