Semantic Belief-State World Model for 3D Human Motion Prediction

TL;DR

This work reframes human motion prediction as latent dynamical simulation by introducing the Semantic Belief-State World Model (SBWM), which maintains a persistent belief state on the SMPL-X body manifold and treats pose as an emission from latent dynamics. By decoupling observation from dynamics and enforcing semantic alignment with SMPL-X, SBWM achieves coherent long-horizon rollouts, multi-modal uncertainty, and improved stability with lower computational cost than diffusion or transformer-based methods. The model demonstrates strong long-horizon performance on 3DPW with favorable efficiency, and provides detailed analyses of failure modes, latent usage, and uncertainty calibration. Overall, SBWM represents a principled shift from reconstruction-focused sequence models to world-model-style latent simulation for human motion, with implications for embodied AI, robotics, and planning.

Abstract

Human motion prediction has traditionally been framed as a sequence regression problem where models extrapolate future joint coordinates from observed pose histories. While effective over short horizons this approach does not separate observation reconstruction with dynamics modeling and offers no explicit representation of the latent causes governing motion. As a result, existing methods exhibit compounding drift, mean-pose collapse, and poorly calibrated uncertainty when rolled forward beyond the training regime. Here we propose a Semantic Belief-State World Model (SBWM) that reframes human motion prediction as latent dynamical simulation on the human body manifold. Rather than predicting poses directly, SBWM maintains a recurrent probabilistic belief state whose evolution is learned independently of pose reconstruction and explicitly aligned with the SMPL-X anatomical parameterization. This alignment imposes a structural information bottleneck that prevents the latent state from encoding static geometry or sensor noise, forcing it to capture motion dynamics, intent, and control-relevant structure. Inspired by belief-state world models developed for model-based reinforcement learning, SBWM adapts stochastic latent transitions and rollout-centric training to the domain of human motion. In contrast to RSSM-based, transformer, and diffusion approaches optimized for reconstruction fidelity, SBWM prioritizes stable forward simulation. We demonstrate coherent long-horizon rollouts, and competitive accuracy at substantially lower computational cost. These results suggest that treating the human body as part of the world models state space rather than its output fundamentally changes how motion is simulated, and predicted.

Figures (10)

• Figure 1: Conceptual Overview of the Semantic Belief-State World Model (SBWM).(A) During observation, the model encodes noisy pose history $x_t$ into a recurrent belief state $h_t$. (B) Once observations cease, the model switches to pure simulation mode. (C) Future dynamics are governed by the interaction between the deterministic belief state $h_t$ and stochastic latent variables $z_t$. Unlike standard autoregressive models that feed back predictions (causing drift), SBWM evolves purely in the latent space, decoding outputs $\hat{x}_t$ only for visualization. This separation enables long-horizon stability on the SMPL-X manifold.
• Figure 2: The SMPL-X body manifold parameterization. Visualizing the semantic landmarks and kinematic structure encoded by the SMPL-X model SMPL-Anthropometry. Unlike raw point clouds, this parametric representation defines a low-dimensional manifold where every coordinate corresponds to a valid anatomical configuration. By constraining our world model to predict these parameters, we enforce geometric consistency and prevent the "limb stretching" artifacts common in joint-based regression.
• Figure 3: Schematic of the Semantic Belief-State World Model. Observations are encoded and used to update a persistent belief state, which evolves under stochastic latent dynamics. Future motion is generated by decoding from the belief state rather than directly regressing poses.
• Figure 4: Evaluation on the 3D Poses in the Wild (3DPW) dataset. A sample frame illustrating the challenging in-the-wild nature of the 3DPW benchmark vonMarcard2018.
• Figure 5: Efficiency-Accuracy Trade-off. SBWM occupies the Pareto-optimal frontier, achieving error rates comparable to computationally expensive Diffusion models while maintaining the real-time latency of lightweight RNNs. Transformers and Joint-space RSSMs suffer from either higher latency or lower accuracy.