A Recurrent Neural Network Enhanced Unscented Kalman Filter for Human Motion Prediction

TL;DR

The paper tackles human arm motion prediction for safe human–robot collaboration in manufacturing by fusing data-driven and physics-based modeling within an adaptive UKF. It introduces two RNNs: one generating a preliminary bone-vector pose prediction used as a surrogate measurement, and another predicting muscle forces; both are augmented with Monte Carlo dropout to quantify uncertainty and adapt the UKF covariances in real time. The arm is modeled using Lagrangian dynamics $F=M(q)\\ddot{q}+C(q,\\dot{q})+G(q)$ with generalized coordinates $q=[\\phi_1;\\theta_1;\\phi_2;\\theta_2]$, linking motions to muscle forces, which improves future motion forecasts via the UKF. Experimental results show enhanced accuracy and robustness, particularly for dynamic motions, while reducing the tuning burden through uncertainty-driven covariance adaptation, highlighting practical benefits for safe human-robot collaboration. $^{1}$ The approach integrates physics-informed intelligence with probabilistic neural prediction to deliver reliable, uncertainty-aware motion forecasts suitable for real-time robot assistance and safety planning. $^{}$

Abstract

This paper presents a deep learning enhanced adaptive unscented Kalman filter (UKF) for predicting human arm motion in the context of manufacturing. Unlike previous network-based methods that solely rely on captured human motion data, which is represented as bone vectors in this paper, we incorporate a human arm dynamic model into the motion prediction algorithm and use the UKF to iteratively forecast human arm motions. Specifically, a Lagrangian-mechanics-based physical model is employed to correlate arm motions with associated muscle forces. Then a Recurrent Neural Network (RNN) is integrated into the framework to predict future muscle forces, which are transferred back to future arm motions based on the dynamic model. Given the absence of measurement data for future human motions that can be input into the UKF to update the state, we integrate another RNN to directly predict human future motions and treat the prediction as surrogate measurement data fed into the UKF. A noteworthy aspect of this study involves the quantification of uncertainties associated with both the data-driven and physical models in one unified framework. These quantified uncertainties are used to dynamically adapt the measurement and process noises of the UKF over time. This adaption, driven by the uncertainties of the RNN models, addresses inaccuracies stemming from the data-driven model and mitigates discrepancies between the assumed and true physical models, ultimately enhancing the accuracy and robustness of our predictions. Compared to the traditional RNN-based prediction, our method demonstrates improved accuracy and robustness in extensive experimental validations of various types of human motions.

Figures (7)

• Figure 1: Overview of the proposed method: (1) We convert the observed motion into bone vectors and employ the prediction model A along with specific kinematic constraints like bone length to generate the preliminary prediction. The preliminary prediction serves as the measurement data of UKF. (2) Observed muscle forces are calculated based on the arm dynamic model, and the prediction model B is utilized to generate future muscle forces acting on the shoulder and elbow joints. (3) We quantify uncertainties of the prediction models A and B, and dynamically adjust the measurement and process noise covariances of UKF using the quantified uncertainties. (4) UKF eventually outputs the refined prediction, and the red shadow areas indicate uncertainties of refined motions.
• Figure 2: Uncertainty quantification: The observed arm poses are represented using blue color, the predicted arm poses are represented using purple color. Using Monte Carlo dropout sampling method, a single observed sequence can generate multiple predicted sequences. The predictive distribution is generated based on the motion statistics from the sample bin. Each step contains multiple possible arm poses.
• Figure 3: Human arm model: The arm motion is tracked using $\phi_1$, $\theta_1$, $\phi_2$, and $\theta_2$. Additionally, $\phi_1$ is the angle between $a_3$ and the vector of upper-arm $s^a$, $\theta_1$ is the angle between $a_1$ and the projection vector of $s^a$, $\phi_2$ is the angle between $b_3$ and the vector of forearm $s^b$, and $\theta_2$ is the angle between $b_1$ and the projection vector of $s^b$.
• Figure 4: Illustration of three human motions: The numbers 1, 2, and 3 indicates the key positions of human hand. In motion A, the human worker first moves forward to grab a screwdriver in the toolbox, then moves back to the start position on the desk. In motion B, the human worker initially grabs a screwdriver on the left side, then moves back to the start position. In motion C, the human worker's first action is to pick up the screwdriver on the desk, followed by placing it into the toolbox, and ultimately returning to the start position.
• Figure 5: Prediction error of elbow using BV and BV-AUKF for each motion category. Each motion sample has 50 steps of prediction.