Beyond Gait: Learning Knee Angle for Seamless Prosthesis Control in Multiple Scenarios

TL;DR

This work addresses knee-angle estimation for prosthesis control beyond gait cycles by introducing AEPM, a transformer-based probabilistic model that leverages whole-body joint movements to predict knee motion from partial poses. AEPM uses a MixSTE-inspired encoder and three decoders to output a mean, variance, and samples, enabling reconstruction of full poses and uncertainty-aware knee-angle predictions. Across Human3.6M and CMU mocap datasets, AEPM achieves an overall RMSE of $6.70^\circ$ and $3.45^\circ$ in walking, outperforming several state-of-the-art methods and enabling seamless transitions between locomotion modes. The study highlights the value of whole-body information for knee dynamics and provides insights into joint synergy and sensor design, with future work aiming for end-to-end systems and amputee validation.

Abstract

Deep learning models have become a powerful tool in knee angle estimation for lower limb prostheses, owing to their adaptability across various gait phases and locomotion modes. Current methods utilize Multi-Layer Perceptrons (MLP), Long-Short Term Memory Networks (LSTM), and Convolutional Neural Networks (CNN), predominantly analyzing motion information from the thigh. Contrary to these approaches, our study introduces a holistic perspective by integrating whole-body movements as inputs. We propose a transformer-based probabilistic framework, termed the Angle Estimation Probabilistic Model (AEPM), that offers precise angle estimations across extensive scenarios beyond walking. AEPM achieves an overall RMSE of 6.70 degrees, with an RMSE of 3.45 degrees in walking scenarios. Compared to the state of the art, AEPM has improved the prediction accuracy for walking by 11.31%. Our method can achieve seamless adaptation between different locomotion modes. Also, this model can be utilized to analyze the synergy between the knee and other joints. We reveal that the whole body movement has valuable information for knee movement, which can provide insights into designing sensors for prostheses. The code is available at https://github.com/penway/Beyond-Gait-AEPM.

Figures (6)

• Figure 1: Overview of the proposed method, and potential pipeline when integrated in the control system. Our method reconstructs the partial poses into full poses
• Figure 2: Detailed structure of Angle Estimation Probabilistic Model (AEPM). The encoder structure mainly consists of a stack of spatial and temporal transformer blocks, basically following the setting of vaswani2017attentionzhang2022mixste. The spatial and temporal self-attention apply the self-attention on spatial and temporal dimensions respectively. The decoders are all Multi-Layer Perceptron which share a similar structure, consisting of three linear layers.
• Figure 3: Time-domain results of the knee angle prediction by AEPM. The red line represents the ground truth, the dark blue line represents the mean of predictions and the light blue lines are all the predictions. The graph titles specify the motion type, file reference, and original frame range from the datasets: Human 3.6M (a,b,f-i) and CMU mocap database (c-e). The accuracy of AEPM in estimating knee angles is demonstrated across a spectrum of motion types, including both rhythmic and non-rhythmic activities, as well as walking and other types of motion.
• Figure 4: The distribution of attention weights across four layers of spatial transformer blocks in a walking scenario in Human 3.6M, with the right knee (masked) designated as the query point. With the progression in deeper layers, there is a discernible shift in attention from joints in close proximity such as the thigh, other leg, and hip, to more distant joints, for instance, the shoulders. Notably, in the final layer, the focus intensifies on the joint under prediction.
• Figure 5: The distribution and progression of attention weights in the first layers of spatial transformer block in a gait cycle with the right knee (masked) designated as the query point. Graph (a) illustrates the reference of ground truth and predicted knee angle in this gait cycle. Graph (b) shows the attention weights distribution. The relative attention weights remain similar across the whole gait cycle, showing that the basic focus of the model remains similar, while there are also cyclical changes, indicating the model is also adjusting the focus based on different poses in human movement.