Personalization of Wearable Sensor-Based Joint Kinematic Estimation Using Computer Vision for Hip Exoskeleton Applications

TL;DR

A computer vision-based DL adaptation framework for realtime joint kinematics estimation that demonstrates a potential for smartphone camera-trained DL model to estimate real-time joint kinematics across novel users in clinical populations with applications in wearable robots.

Abstract

Accurate lower-limb joint kinematic estimation is critical for applications such as patient monitoring, rehabilitation, and exoskeleton control. While previous studies have employed wearable sensor-based deep learning (DL) models for estimating joint kinematics, these methods often require extensive new datasets to adapt to unseen gait patterns. Meanwhile, researchers in computer vision have advanced human pose estimation models, which are easy to deploy and capable of real-time inference. However, such models are infeasible in scenarios where cameras cannot be used. To address these limitations, we propose a computer vision-based DL adaptation framework for real-time joint kinematic estimation. This framework requires only a small dataset (i.e., 1-2 gait cycles) and does not depend on professional motion capture setups. Using transfer learning, we adapted our temporal convolutional network (TCN) to stiff knee gait data, allowing the model to further reduce root mean square error by 9.7% and 19.9% compared to a TCN trained on only able-bodied and stiff knee datasets, respectively. Our framework demonstrates a potential for smartphone camera-trained DL models to estimate real-time joint kinematics across novel users in clinical populations with applications in wearable robots.

Figures (7)

• Figure 2: Wearable sensing suit for joint angle estimation. (a) Components of the sensing suit hardware. (b) Data flow within the sensing suit for sensor data logging and real-time inference of joint kinematics.
• Figure 3: Pose estimation pipeline using a human vision model. (a) Overall pipeline for extracting 3D joint keypoints from video frames. (b) Visual diagram depicting how joint angles are measured from the 3D keypoints. (c) Plot of estimated joint angles (solid lines) for an able-bodied subject walking at 1.0 m/s, generated using the pose estimation pipeline, compared to the ground-truth joint angles (dotted lines).
• Figure 4: Motion capture setup for measuring ground-truth joint angles and vision model-calculated joint angles.
• Figure 5: Structure of the Temporal Convolutional Network (TCN) for training the kinematic estimation model.
• Figure 6: Experimental setup for evaluating stiff-knee (SK) behavior and the estimation error of a transfer-learned, adapted machine learning model. (a) Knee brace used to replicate the SK gait pattern. (b) Ground-truth joint angle trajectories during SK gait for a single subject walking at 1 m/s. (c) Comparison of estimation errors between the SK model and the adapted AB+SK model across varying SK-to-AB dataset ratios (mean ± SD, three SK subjects).