Differentiable Biomechanics Unlocks Opportunities for Markerless Motion Capture

TL;DR

This work demonstrates a GPU-accelerated, differentiable biomechanics framework for markerless motion capture that jointly estimates skeleton scaling, marker offsets, and joint trajectories by propagating implicit pose representations through a forward kinematic model. It introduces end-to-end bilevel optimization, bundle adjustment of camera parameters, and a tri-level meta-optimization across multiple individuals to improve reprojection accuracy and gait metrics. The approach yields tighter geometric consistency with 2D keypoints and robust step-parameter estimates across a diverse participant cohort, including clinical populations, and highlights the potential to unify computer vision and biomechanics in a scalable, differentiable pipeline. These advances enable more accurate, personalized biomechanical analyses from markerless data and pave the way for uncertainty-aware, hands- and multi-person capable gait analysis systems.

Abstract

Recent developments have created differentiable physics simulators designed for machine learning pipelines that can be accelerated on a GPU. While these can simulate biomechanical models, these opportunities have not been exploited for biomechanics research or markerless motion capture. We show that these simulators can be used to fit inverse kinematics to markerless motion capture data, including scaling the model to fit the anthropomorphic measurements of an individual. This is performed end-to-end with an implicit representation of the movement trajectory, which is propagated through the forward kinematic model to minimize the error from the 3D markers reprojected into the images. The differential optimizer yields other opportunities, such as adding bundle adjustment during trajectory optimization to refine the extrinsic camera parameters or meta-optimization to improve the base model jointly over trajectories from multiple participants. This approach improves the reprojection error from markerless motion capture over prior methods and produces accurate spatial step parameters compared to an instrumented walkway for control and clinical populations.

Figures (6)

• Figure 1: Visualization of reprojection errors. Each panel is a zoomed in crop from the image around the detected bounding box from a different view. The red dots are the 87 detected keypoints in each frame and the blue dots are the reprojected model-based markers, with the drawn skeleton connecting a subset of these keypoints.
• Figure 2: Sample rendered frames from a walking trial fit from an individual with a history of stroke using a cane on the left side. The top row is the modified LocoMujoco skeleton and the bottom row is the MyoSkeleton.
• Figure 3: Sample waveforms from a control participant. The left leg is shown in blue and the right leg is shown in red. The top row shows the lateral position of the heel with the positions detected from the instrumented walkway shown in black. The next row shows the vertical clearance from the heel and toes. The last three rows show the flexion angles for the hip, knee, and ankle, respectively.
• Figure 4: Sample waveforms from a participant with a history of stroke, corresponding to the visualization from Figure \ref{['fig:walking_frames']}. This captures the reduced range of motion on the right hemiparetic side, particularly with reduced knee flexion.
• Figure 5: Reprojection losses for reconstruction with two-stage approach using implicit representation followed by nimblephysics versus end-to-end approach with implicit representation and forward kinematic biomechanical model in mujoco. The left panel compares the $GC_5$ between our prior two-stage approach using nimble physics (horizontal axis) with differentiable biomechanics (vertical axis) for a number of trials. The middle panel compares trials with and without bundle adjustment to refine calibration. The right panel shows the average $GC_x$ as a function of the pixel threshold for those trials.