Human Motion Prediction under Unexpected Perturbation

TL;DR

The paper tackles predicting human motion under unexpected perturbations, including how perturbations propagate across groups, by introducing Latent Differentiable Physics (LDP). LDP combines a latent, differentiable IPM core with a skeleton-restoration network, enabling data-efficient learning of balance recovery and inter-person interactions, while providing interpretable explanations via learned forces. The method achieves state-of-the-art or near-best performance across single- and multi-person perturbation scenarios on the FZJ Push dataset, and demonstrates strong generalization to unseen poses, magnitudes, timings, and larger group configurations. A key advance is explicit modeling of physical interactions in a low-dimensional latent space, which yields high data efficiency, robust generalization, and explainability, albeit with some trade-offs in generality relative to fully black-box approaches. Overall, LDP offers a practical framework for reactive motion prediction in crowded or perturbation-heavy settings, with potential applications in biomechanics, crowd simulation, and human-robot collaboration.

Abstract

We investigate a new task in human motion prediction, which is predicting motions under unexpected physical perturbation potentially involving multiple people. Compared with existing research, this task involves predicting less controlled, unpremeditated and pure reactive motions in response to external impact and how such motions can propagate through people. It brings new challenges such as data scarcity and predicting complex interactions. To this end, we propose a new method capitalizing differential physics and deep neural networks, leading to an explicit Latent Differential Physics (LDP) model. Through experiments, we demonstrate that LDP has high data efficiency, outstanding prediction accuracy, strong generalizability and good explainability. Since there is no similar research, a comprehensive comparison with 11 adapted baselines from several relevant domains is conducted, showing LDP outperforming existing research both quantitatively and qualitatively, improving prediction accuracy by as much as 70%, and demonstrating significantly stronger generalization.

Figures (16)

• Figure 1: Overview of our model. Given a frame $X_t$, it is first mapped into the IPM space via Skeleton-to-IPM to get its IPM state $I_t$. Then $I_t$ is simulated for one step via Differentiable IPM to compute $I_{t+1}$. Lastly, the full-body frame $X_{t+1}$ is recovered from $I_{t+1}$ via Skeleton Restoration Model. The IPM is shown in the right figure. The full-body state $X$ is represented by joint positions.
• Figure 2: FZJ Push feldmann2023forward. The blue agent was pushed by the punch bag and then he pushed other people.
• Figure 3: Perturbations with different magnitudes in single-person (top) and multi-people (bottom).
• Figure 4: Visual Results in the Single-person scenario.
• Figure 5: Multi-people comparison. The last row shows the learned net force on the second (from the left) person. The bar height indicates the magnitude and the sign indicates the direction, where the people move in the positive direction of the x-axis.