Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OpenCapBench: A Benchmark to Bridge Pose Estimation and Biomechanics

Yoni Gozlan, Antoine Falisse, Scott Uhlrich, Anthony Gatti, Michael Black, Akshay Chaudhari

TL;DR

OpenCapBench addresses the gap between pose estimation benchmarks and biomechanics by providing a physiologically constrained, end-to-end framework that converts pose outputs into OpenSim-based joint kinematics and a public leaderboard. It introduces SynthPose, a synthetic-data–driven finetuning approach to predict a denser, anatomically meaningful set of 2D keypoints, improving biomechanical accuracy and reducing joint-angle RMSE. Through comprehensive experiments on OpenCapBench and RICH, the work demonstrates substantial gains over sparse-keypoint approaches and SMPL-based methods, highlighting the value of biomechanical metrics for evaluating pose estimation. By enabling researchers to benchmark new methods with clinically relevant kinematic criteria, OpenCapBench has the potential to accelerate advances in movement analysis and health applications.

Abstract

Pose estimation has promised to impact healthcare by enabling more practical methods to quantify nuances of human movement and biomechanics. However, despite the inherent connection between pose estimation and biomechanics, these disciplines have largely remained disparate. For example, most current pose estimation benchmarks use metrics such as Mean Per Joint Position Error, Percentage of Correct Keypoints, or mean Average Precision to assess performance, without quantifying kinematic and physiological correctness - key aspects for biomechanics. To alleviate this challenge, we develop OpenCapBench to offer an easy-to-use unified benchmark to assess common tasks in human pose estimation, evaluated under physiological constraints. OpenCapBench computes consistent kinematic metrics through joints angles provided by an open-source musculoskeletal modeling software (OpenSim). Through OpenCapBench, we demonstrate that current pose estimation models use keypoints that are too sparse for accurate biomechanics analysis. To mitigate this challenge, we introduce SynthPose, a new approach that enables finetuning of pre-trained 2D human pose models to predict an arbitrarily denser set of keypoints for accurate kinematic analysis through the use of synthetic data. Incorporating such finetuning on synthetic data of prior models leads to twofold reduced joint angle errors. Moreover, OpenCapBench allows users to benchmark their own developed models on our clinically relevant cohort. Overall, OpenCapBench bridges the computer vision and biomechanics communities, aiming to drive simultaneous advances in both areas.

OpenCapBench: A Benchmark to Bridge Pose Estimation and Biomechanics

TL;DR

OpenCapBench addresses the gap between pose estimation benchmarks and biomechanics by providing a physiologically constrained, end-to-end framework that converts pose outputs into OpenSim-based joint kinematics and a public leaderboard. It introduces SynthPose, a synthetic-data–driven finetuning approach to predict a denser, anatomically meaningful set of 2D keypoints, improving biomechanical accuracy and reducing joint-angle RMSE. Through comprehensive experiments on OpenCapBench and RICH, the work demonstrates substantial gains over sparse-keypoint approaches and SMPL-based methods, highlighting the value of biomechanical metrics for evaluating pose estimation. By enabling researchers to benchmark new methods with clinically relevant kinematic criteria, OpenCapBench has the potential to accelerate advances in movement analysis and health applications.

Abstract

Pose estimation has promised to impact healthcare by enabling more practical methods to quantify nuances of human movement and biomechanics. However, despite the inherent connection between pose estimation and biomechanics, these disciplines have largely remained disparate. For example, most current pose estimation benchmarks use metrics such as Mean Per Joint Position Error, Percentage of Correct Keypoints, or mean Average Precision to assess performance, without quantifying kinematic and physiological correctness - key aspects for biomechanics. To alleviate this challenge, we develop OpenCapBench to offer an easy-to-use unified benchmark to assess common tasks in human pose estimation, evaluated under physiological constraints. OpenCapBench computes consistent kinematic metrics through joints angles provided by an open-source musculoskeletal modeling software (OpenSim). Through OpenCapBench, we demonstrate that current pose estimation models use keypoints that are too sparse for accurate biomechanics analysis. To mitigate this challenge, we introduce SynthPose, a new approach that enables finetuning of pre-trained 2D human pose models to predict an arbitrarily denser set of keypoints for accurate kinematic analysis through the use of synthetic data. Incorporating such finetuning on synthetic data of prior models leads to twofold reduced joint angle errors. Moreover, OpenCapBench allows users to benchmark their own developed models on our clinically relevant cohort. Overall, OpenCapBench bridges the computer vision and biomechanics communities, aiming to drive simultaneous advances in both areas.
Paper Structure (19 sections, 1 equation, 5 figures, 5 tables)

This paper contains 19 sections, 1 equation, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. OpenCapBench Pipeline
  4. Dataset Integration
  5. Benchmarking Pipeline
  6. 2D keypoints extraction
  7. 3D Triangulation
  8. Inverse Kinematics (IK)
  9. Evaluation Metrics
  10. Leaderboard
  11. Arbitrary 2D keypoints estimation using synthetic data
  12. Motivation
  13. Leveraging synthetic data to finetune pose estimation models
  14. Experiments
  15. New arbitrary 2D keypoints prediction method
  16. ...and 4 more sections

Figures (5)

  • Figure 1: OpenCapBench pipeline. SynthPose, our method to finetune 2D pose estimation models to predict any set of body keypoints (designated by a star here) is detailed in \ref{['fig:finetuning']}.
  • Figure 2: Chosen subset of 35 vertices from SMPL mesh.
  • Figure 3: Extracting 2D keypoints from SMPL mesh.
  • Figure 4: SynthPose: a new method leveraging finetuning of pose estimation models on synthetic data to predict an arbitrary set of 2D keypoints.
  • Figure 5: Samples of Synthpose MoCap marker predictions on each subject of the OpenCap dataset, with their corresponding OpenSim kinematics output. The pink markers represent anatomical markers on the right side of the body, the cyan markers represent those on the left side, the blue markers represent markers in the center of the body, and the red markers represent markers in the COCO format. The SynthPose model used is HRNet48 fine-tuned on the entire aggregated dataset to predict the subset of markers defined in the main submission.