Improving 3D Foot Motion Reconstruction in Markerless Monocular Human Motion Capture

TL;DR

Experiments show that FootMR outperforms state-of-the-art methods, reducing ankle joint angle error on MOYO by up to 30% over the best video-based approach and introducing MOOF, a 2D dataset of complex foot movements.

Abstract

State-of-the-art methods can recover accurate overall 3D human body motion from in-the-wild videos. However, they often fail to capture fine-grained articulations, especially in the feet, which are critical for applications such as gait analysis and animation. This limitation results from training datasets with inaccurate foot annotations and limited foot motion diversity. We address this gap with FootMR, a Foot Motion Refinement method that refines foot motion estimated by an existing human recovery model through lifting 2D foot keypoint sequences to 3D. By avoiding direct image input, FootMR circumvents inaccurate image-3D annotation pairs and can instead leverage large-scale motion capture data. To resolve ambiguities of 2D-to-3D lifting, FootMR incorporates knee and foot motion as context and predicts only residual foot motion. Generalization to extreme foot poses is further improved by representing joints in global rather than parent-relative rotations and applying extensive data augmentation. To support evaluation of foot motion reconstruction, we introduce MOOF, a 2D dataset of complex foot movements. Experiments on MOOF, MOYO, and RICH show that FootMR outperforms state-of-the-art methods, reducing ankle joint angle error on MOYO by up to 30% over the best video-based approach.

Figures (6)

• Figure 1: State-of-the-art 3D human motion recovery methods like GVHMRshen2024gvhmr(top) fail to capture complex 3D foot movement when given in-the-wild videos. We identify this to be mainly an issue of inaccurate and insufficient video training data. To address this, we introduce FootMR, a Foot Motion Refinement method that leverages large-scale motion capture data to learn lifting 2D foot keypoint sequences to 3D. By effectively resolving ambiguities in 2D-to-3D mapping, FootMR(bottom), when combined with an existing 3D human recovery model, generates realistic and accurate 3D foot motion, significantly outperforming previous work.
• Figure 2: Erroneous 3D foot annotations in pseudo-GT fits generated by fitting 3D models to sparse keypoints. Images are from MPII andriluka2014mpii, COCO lin2014coco, and 3DPW marcard2018eccv. Please zoom in for details.
• Figure 3: Overview of our framework. Given a monocular video, a SMPL-X motion estimator is employed to estimate 3D human motion including erroneous initial ankle rotations. Knee and ankle predictions are then transformed from parent-relative to global rotations and used together with 2D foot keypoints and bounding boxes as input for Foot Motion Refinement (FootMR). FootMR refines the initial ankle predictions by estimating residual rotations. After fusing the refined ankle rotations with the remaining body parameters, the final output of our framework is accurate and temporally coherent 3D human body and foot motion.
• Figure 4: Qualitative comparison on the introduced MOOF dataset. FootMR is the only method that is able to accurately reconstruct the extreme foot poses.
• Figure S1: Video frames from MOOF. The MOOF dataset consists of videos with annotated 2D keypoints for the left and right big toe, small toe, and heel. The collected videos are designed to highlight complex foot movements, making the dataset well-suited for evaluating fine-grained foot motion reconstruction in scenarios such as dance, sports, and rehabilitation.