Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spatiotemporal-Untrammelled Mixture of Experts for Multi-Person Motion Prediction

Zheng Yin, Chengjian Li, Xiangbo Shu, Meiqi Cao, Rui Yan, Jinhui Tang

TL;DR

<3-5 sentence high-level summary>ST-MoE tackles multi-person motion prediction by addressing two key weaknesses of prior Transformer-based methods: rigid spatiotemporal representations and the high cost of attention. It introduces a Mixture of Spatiotemporal Mamba Experts framework with four specialized bidirectional Mamba blocks and a dynamic routing mechanism, enabling flexible, linearly-scaling modeling of complex spatiotemporal dependencies. The approach achieves state-of-the-art accuracy across CMU-Mocap(UMPM), Mix1, Mix2, and CHI3D while reducing parameter count by 41.38% and accelerating training by 3.6×, demonstrating strong practical impact for efficient MPMP. The work integrates DCT/iDCT preprocessing, a GCN-based encoder/decoder, and a balanced loss to robustly capture dynamics, with future work extending to stochastic predictions and broader MoE configurations.

Abstract

Comprehensively and flexibly capturing the complex spatio-temporal dependencies of human motion is critical for multi-person motion prediction. Existing methods grapple with two primary limitations: i) Inflexible spatiotemporal representation due to reliance on positional encodings for capturing spatiotemporal information. ii) High computational costs stemming from the quadratic time complexity of conventional attention mechanisms. To overcome these limitations, we propose the Spatiotemporal-Untrammelled Mixture of Experts (ST-MoE), which flexibly explores complex spatio-temporal dependencies in human motion and significantly reduces computational cost. To adaptively mine complex spatio-temporal patterns from human motion, our model incorporates four distinct types of spatiotemporal experts, each specializing in capturing different spatial or temporal dependencies. To reduce the potential computational overhead while integrating multiple experts, we introduce bidirectional spatiotemporal Mamba as experts, each sharing bidirectional temporal and spatial Mamba in distinct combinations to achieve model efficiency and parameter economy. Extensive experiments on four multi-person benchmark datasets demonstrate that our approach not only outperforms state-of-art in accuracy but also reduces model parameter by 41.38% and achieves a 3.6x speedup in training. The code is available at https://github.com/alanyz106/ST-MoE.

Spatiotemporal-Untrammelled Mixture of Experts for Multi-Person Motion Prediction

TL;DR

<3-5 sentence high-level summary>ST-MoE tackles multi-person motion prediction by addressing two key weaknesses of prior Transformer-based methods: rigid spatiotemporal representations and the high cost of attention. It introduces a Mixture of Spatiotemporal Mamba Experts framework with four specialized bidirectional Mamba blocks and a dynamic routing mechanism, enabling flexible, linearly-scaling modeling of complex spatiotemporal dependencies. The approach achieves state-of-the-art accuracy across CMU-Mocap(UMPM), Mix1, Mix2, and CHI3D while reducing parameter count by 41.38% and accelerating training by 3.6×, demonstrating strong practical impact for efficient MPMP. The work integrates DCT/iDCT preprocessing, a GCN-based encoder/decoder, and a balanced loss to robustly capture dynamics, with future work extending to stochastic predictions and broader MoE configurations.

Abstract

Comprehensively and flexibly capturing the complex spatio-temporal dependencies of human motion is critical for multi-person motion prediction. Existing methods grapple with two primary limitations: i) Inflexible spatiotemporal representation due to reliance on positional encodings for capturing spatiotemporal information. ii) High computational costs stemming from the quadratic time complexity of conventional attention mechanisms. To overcome these limitations, we propose the Spatiotemporal-Untrammelled Mixture of Experts (ST-MoE), which flexibly explores complex spatio-temporal dependencies in human motion and significantly reduces computational cost. To adaptively mine complex spatio-temporal patterns from human motion, our model incorporates four distinct types of spatiotemporal experts, each specializing in capturing different spatial or temporal dependencies. To reduce the potential computational overhead while integrating multiple experts, we introduce bidirectional spatiotemporal Mamba as experts, each sharing bidirectional temporal and spatial Mamba in distinct combinations to achieve model efficiency and parameter economy. Extensive experiments on four multi-person benchmark datasets demonstrate that our approach not only outperforms state-of-art in accuracy but also reduces model parameter by 41.38% and achieves a 3.6x speedup in training. The code is available at https://github.com/alanyz106/ST-MoE.
Paper Structure (53 sections, 11 equations, 7 figures, 7 tables, 2 algorithms)

This paper contains 53 sections, 11 equations, 7 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Human Motion Prediction.
  4. Mixture of Experts.
  5. State Space Model.
  6. Method
  7. Overview
  8. Problem Statement
  9. Multi-Pose Encoder and Decoder
  10. Mixture of Spatiotemporal Mamba Experts
  11. Bidirectional Spatiotemporal Mamba
  12. Loss Function
  13. Experiments
  14. Datasets
  15. CMU-Mocap (UMPM).
  16. ...and 38 more sections

Figures (7)

  • Figure 1: Insight of our work and performance of models on dataset CMU-Mocap(UMPM). (a)-(c): Limitations of prior methods using attention mechanism/spatio-temporal positional encodings, which solely capture trammelled spatiotemporal patterns. TPE/SPE denote temporal/spatial positional encoding. TRPE denotes trajectory-aware relative position encoding. (d): Our MoE-based framework with dynamic expert selection for adaptive spatiotemporal modeling. (e): Our model achieves an optimal efficiency-performance trade-off. For fair comparison, the batch size of all models was set to 96.
  • Figure 2: Overview of Spatiotemporal-Untrammelled Mixture of Experts (ST-MoE). The padded input motion sequence is first encoded and then adaptively routed by a gating router to distinct spatiotemporal experts. ST-MoE aggregates the outputs of the selected experts and decodes the merged features to predict the future motion sequences. Each expert in the expert pool consists of pairwise combinations of bidirectional temporal Mamba and bidirectional spatial Mamba, with shared parameters across all experts for both temporal and spatial Mamba components. This design enables comprehensive learning of spatiotemporal dependencies in human motion while maintaining lightweight.
  • Figure 3: (a): Impact of different numbers of activated experts. (b): Impact of different numbers of MoE layers. Both are tested on the CMU-Mocap (UMPM) dataset.
  • Figure 4: Visualization comparison on CMU-Mocap (UMPM) dataset. Dark-colored lines represent predicted motion, while light-colored lines indicate ground truth.
  • Figure 5: The t-SNE visualization of features learned by four distinct experts on CHI3D and Mix2 dataset. 300 samples are randomly selected. Best view in color.
  • ...and 2 more figures