VidSole: A Multimodal Dataset for Joint Kinetics Quantification and Disease Detection with Deep Learning

TL;DR

VidSole tackles the challenge of scalable, accessible gait analysis for knee osteoarthritis by delivering a multimodal dataset that fuses instrumented insoles, two-view RGB video, motion capture, and force plate data for over $2{,}600$ trials from $52$ participants. A two-stage deep learning pipeline first classifies activities with a GRU ensemble and then estimates knee adduction moment ($KAM$) with activity-specific LSTM regressors, achieving a mean absolute error below the clinically meaningful threshold of $0.5\%\,BW\cdot ht$ and a walking correlation of $r=0.94$. The results underscore the advantages of integrating insole shear/moments with pose-derived kinematics, outperforming existing video-only or mocap-based approaches. By providing a publicly available, richly annotated multimodal dataset and baseline models, the work paves the way for non-lab, data-driven gait analysis that could aid early OA detection and broader biomechanical assessments in clinical settings.

Abstract

Understanding internal joint loading is critical for diagnosing gait-related diseases such as knee osteoarthritis; however, current methods of measuring joint risk factors are time-consuming, expensive, and restricted to lab settings. In this paper, we enable the large-scale, cost-effective biomechanical analysis of joint loading via three key contributions: the development and deployment of novel instrumented insoles, the creation of a large multimodal biomechanics dataset (VidSole), and a baseline deep learning pipeline to predict internal joint loading factors. Our novel instrumented insole measures the tri-axial forces and moments across five high-pressure points under the foot. VidSole consists of the forces and moments measured by these insoles along with corresponding RGB video from two viewpoints, 3D body motion capture, and force plate data for over 2,600 trials of 52 diverse participants performing four fundamental activities of daily living (sit-to-stand, stand-to-sit, walking, and running). We feed the insole data and kinematic parameters extractable from video (i.e., pose, knee angle) into a deep learning pipeline consisting of an ensemble Gated Recurrent Unit (GRU) activity classifier followed by activity-specific Long Short Term Memory (LSTM) regression networks to estimate knee adduction moment (KAM), a biomechanical risk factor for knee osteoarthritis. The successful classification of activities at an accuracy of 99.02 percent and KAM estimation with mean absolute error (MAE) less than 0.5 percent*body weight*height, the current threshold for accurately detecting knee osteoarthritis with KAM, illustrates the usefulness of our dataset for future research and clinical settings.

Figures (4)

• Figure 1: We assemble VidSole, a dataset that includes RGB video, motion capture, force plate, and instrumented insole pressure forces and moments data. This figure shows the RGB video data, motion capture visualized skeleton, and insole sensor raw force data for a participant walking.
• Figure 2: A) An individual sensor with a thin layer of cork increases its total height to 8mm. B) Sensors are aligned in the cork insole under the toe, medial ball, central ball, lateral ball, and heel. C) Axis of sensor orientation in the insole. The insole comprises two 4 mm pieces of cork and one 2 mm piece of cork. D) Insole in the shoe with Raspberry Pi and custom PCB housed on the top of the shoe.
• Figure 3: Our deep learning pipeline: The insole data and MediaPipe pose estimation extracted from RGB video are used to classify activities using an ensemble Gated Recurrent Unit (GRU) model. After each activity is classified, the insole sensor data and knee angle data are used as inputs to an activity-specific Long Short Term Memory (LSTM) model to predict KAM. Ground truth KAM is calculated via inverse dynamics from the force plate and motion capture data.
• Figure 4: Visualization of mean KAM prediction (red dashed line) and mean reference ground truth (black line) for each activity model for multimodal inputs. One standard deviation is plotted in the shaded area.