FootFormer: Estimating Stability from Visual Input

TL;DR

FootFormer introduces a cross-modality architecture that directly predicts foot pressure distributions, foot contact maps, and 3D center of mass from video-based pose sequences. By employing a GCN-based pose encoder, a spatiotemporal transformer, and cross-attentive decoders, it jointly optimizes three outputs to enable stability analysis. Across PSU-TMM100, MMVP, UnderPressure, and Ordinary Movements, FootFormer achieves state-of-the-art or comparable performance on pressure, contact, and stability metrics, with strong generalization to unseen everyday movements. The work highlights the feasibility of vision-driven stability estimation and provides a unified model that surpasses previous multi-model baselines while offering a public codebase.

Abstract

We propose FootFormer, a cross-modality approach for jointly predicting human motion dynamics directly from visual input. On multiple datasets, FootFormer achieves statistically significantly better or equivalent estimates of foot pressure distributions, foot contact maps, and center of mass (CoM), as compared with existing methods that generate one or two of those measures. Furthermore, FootFormer achieves SOTA performance in estimating stability-predictive components (CoP, CoM, BoS) used in classic kinesiology metrics. Code and data are available at https://github.com/keatonkraiger/Vision-to-Stability.git.

Figures (9)

• Figure 1: The proposed cross-modality architecture FootFormer captures spatiotemporal information from visual input to directly estimate predictive measurements of human dynamics and stability. FootFormer embeds pose sequences and passes them through a spatiotemporal transformer, which is then decoded into a dense foot pressure map, contact estimation, and 3D center of mass (CoM) location, respectively.
• Figure 2: Sample Taiji poses and matching foot pressures measured from PSU-TMM100 (rows 1-2) versus sample Ordinary Movements (rows 3-4) poses and matching foot pressures.
• Figure 3: Foot plantar pressure annotated with Center of Pressure (CoP), Base of Support (BoS), and Center of Mass (CoM) projected onto the floor plane. Two stability metrics are shown: Com-BoS (2D distance from CoM to BoS boundary) and CoM-CoP (2D distance from CoM to CoP).
• Figure 4: Performance comparison on PSU-TMM100 ScottECCV2020. (a,b) Mean and median CoM L$_2$ error across subjects. (c) CoP L$_2$ error and (d) BoS IoU across varying pressure thresholds. FootFormer achieves statistically significantly better results across all stability components (see Supplementary Material for exact p-values).
• Figure 5: Mean KLD and CoM Euclidean error when varying FootFormer's components.