Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MoReFun: Past-Movement Guided Motion Representation Learning for Future Motion Prediction and Understanding

Junyu Shi, Haoting Wu, Zhiyuan Zhang, Lijiang Liu, Yong Sun, Qiang Nie

TL;DR

MoReFun tackles the core issue of representation shortcutting in 3D motion prediction by decoupling representation learning from prediction through a two-stage, self-supervised framework. It introduces Past Movement Guided Pretraining with velocity-based masking to learn rich spatiotemporal priors from past motion, followed by finetuning that jointly predicts future motion and a lightweight future-descriptor, enabling simultaneous low-level prediction and high-level understanding. Across Human3.6M, 3DPW, AMASS, and FineMotion, the approach achieves significant gains in prediction accuracy (e.g., up to 8.8% average MPJPE improvement) and demonstrates competitive performance in future-motion understanding versus LLM-based baselines. The results show that coupling motion prediction with semantic understanding through a lightweight text head yields mutually reinforcing improvements, with a robust architecture built on Past Motion Encoder and Future Motion Decoder employing ST-attention and cross-attention mechanisms.

Abstract

3D human motion prediction aims to generate coherent future motions from observed sequences, yet existing end-to-end regression frameworks often fail to capture complex dynamics and tend to produce temporally inconsistent or static predictions-a limitation rooted in representation shortcutting, where models rely on superficial cues rather than learning meaningful motion structure. We propose a two-stage self-supervised framework that decouples representation learning from prediction. In the pretraining stage, the model performs unified past-future self-reconstruction, reconstructing the past sequence while recovering masked joints in the future sequence under full historical guidance. A velocity-based masking strategy selects highly dynamic joints, forcing the model to focus on informative motion components and internalize the statistical dependencies between past and future states without regression interference. In the fine-tuning stage, the pretrained model predicts the entire future sequence, now treated as fully masked, and is further equipped with a lightweight future-text prediction head for joint optimization of low-level motion prediction and high-level motion understanding. Experiments on Human3.6M, 3DPW, and AMASS show that our method reduces average prediction errors by 8.8% over state-of-the-art methods while achieving competitive future-motion understanding performance compared to LLM-based models. Code is available at: https://github.com/JunyuShi02/MoReFun

MoReFun: Past-Movement Guided Motion Representation Learning for Future Motion Prediction and Understanding

TL;DR

MoReFun tackles the core issue of representation shortcutting in 3D motion prediction by decoupling representation learning from prediction through a two-stage, self-supervised framework. It introduces Past Movement Guided Pretraining with velocity-based masking to learn rich spatiotemporal priors from past motion, followed by finetuning that jointly predicts future motion and a lightweight future-descriptor, enabling simultaneous low-level prediction and high-level understanding. Across Human3.6M, 3DPW, AMASS, and FineMotion, the approach achieves significant gains in prediction accuracy (e.g., up to 8.8% average MPJPE improvement) and demonstrates competitive performance in future-motion understanding versus LLM-based baselines. The results show that coupling motion prediction with semantic understanding through a lightweight text head yields mutually reinforcing improvements, with a robust architecture built on Past Motion Encoder and Future Motion Decoder employing ST-attention and cross-attention mechanisms.

Abstract

3D human motion prediction aims to generate coherent future motions from observed sequences, yet existing end-to-end regression frameworks often fail to capture complex dynamics and tend to produce temporally inconsistent or static predictions-a limitation rooted in representation shortcutting, where models rely on superficial cues rather than learning meaningful motion structure. We propose a two-stage self-supervised framework that decouples representation learning from prediction. In the pretraining stage, the model performs unified past-future self-reconstruction, reconstructing the past sequence while recovering masked joints in the future sequence under full historical guidance. A velocity-based masking strategy selects highly dynamic joints, forcing the model to focus on informative motion components and internalize the statistical dependencies between past and future states without regression interference. In the fine-tuning stage, the pretrained model predicts the entire future sequence, now treated as fully masked, and is further equipped with a lightweight future-text prediction head for joint optimization of low-level motion prediction and high-level motion understanding. Experiments on Human3.6M, 3DPW, and AMASS show that our method reduces average prediction errors by 8.8% over state-of-the-art methods while achieving competitive future-motion understanding performance compared to LLM-based models. Code is available at: https://github.com/JunyuShi02/MoReFun
Paper Structure (49 sections, 20 equations, 12 figures, 13 tables)

This paper contains 49 sections, 20 equations, 12 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. 3D Skeleton-based Human Motion Prediction
  4. Self-supervised Pretraining
  5. Motion Captioning
  6. Method
  7. Problem Background
  8. Model Architecture
  9. Joint Embedder and Predict Head
  10. Past Motion Encoder (PME)
  11. Future Motion Decoder (FMD)
  12. Past Movement Guided Pretraining
  13. Past Motion Reconstruction
  14. Future Motion Reconstruction
  15. Fine-tuning for Prediction and Understanding
  16. ...and 34 more sections

Figures (12)

  • Figure 1: Comparison of complex motion prediction results. While previous methods struggle with complex movements, producing poor and unnatural static poses, our method demonstrates improved accuracy by addressing the inherent weakness in representation capability, yielding a natural motion prediction.
  • Figure 2: Overall architecture of MoReFun, which consists of a Past Motion Encoder (PME) and a Future Motion Decoder (FMD). The PME is built with stacked Spatial and Temporal Self-Attention layers, processing the past motion sequence. The FMD receives masked future motion together with the encoded past-motion features, and is composed of Spatial/Temporal Cross-Attention followed by Spatial/Temporal Self-Attention. Through these stages, the decoder leverages the past-motion representation to guide the reconstruction of masked future joints.
  • Figure 3: Overview of the finetuning stage. The Past Motion Encoder extracts spatiotemporal representations from the observed history motion. The shared past-motion feature is fed into two branches: (1) the Future Motion Decoder, which predicts the future 3D pose sequence from a zero-initialized state, and (2) the Understanding Decoder, which produces a natural-language description of the upcoming motion. Both branches are jointly optimized with motion and text supervision.
  • Figure 4: Visualization results of motion prediction. The red lines represent predicted motions, and the blue lines represent ground-truths.
  • Figure 5: Visualization results of motion reconstruction on H36M. The blue lines represent reconstructed motions, and the red lines represent ground-truths.
  • ...and 7 more figures