Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RePose: A Real-Time 3D Human Pose Estimation and Biomechanical Analysis Framework for Rehabilitation

Junxiao Xue, Pavel Smirnov, Ziao Li, Yunyun Shi, Shi Chen, Xinyi Yin, Xiaohan Yue, Lei Wang, Yiduo Wang, Feng Lin, Yijia Chen, Xiao Ma, Xiaoran Yan, Qing Zhang, Fengjian Xue, Xuecheng Wu

TL;DR

RePose addresses the need for real-time, accurate monitoring and feedback during rehabilitation by presenting an end-to-end multi-camera pipeline that estimates 3D human pose from RGB video. It combines fast single-target tracking, 2D pose estimation, learnable triangulation for 3D reconstruction, and a modified SmoothNet for real-time temporal refinement, augmented with FABRIK-based muscle-force visualization in Unity. The framework achieves approximately 30 FPS on Linux and demonstrates competitive accuracy on the Human3.6M benchmark while enabling real-time muscle-stress visualization to guide patients. This approach holds promise for remote, home-based rehabilitation by delivering immediate guidance, reducing reliance on expensive motion capture, and potentially easing clinical burdens.

Abstract

We propose a real-time 3D human pose estimation and motion analysis method termed RePose for rehabilitation training. It is capable of real-time monitoring and evaluation of patients'motion during rehabilitation, providing immediate feedback and guidance to assist patients in executing rehabilitation exercises correctly. Firstly, we introduce a unified pipeline for end-to-end real-time human pose estimation and motion analysis using RGB video input from multiple cameras which can be applied to the field of rehabilitation training. The pipeline can help to monitor and correct patients'actions, thus aiding them in regaining muscle strength and motor functions. Secondly, we propose a fast tracking method for medical rehabilitation scenarios with multiple-person interference, which requires less than 1ms for tracking for a single frame. Additionally, we modify SmoothNet for real-time posture estimation, effectively reducing pose estimation errors and restoring the patient's true motion state, making it visually smoother. Finally, we use Unity platform for real-time monitoring and evaluation of patients' motion during rehabilitation, and to display the muscle stress conditions to assist patients with their rehabilitation training.

RePose: A Real-Time 3D Human Pose Estimation and Biomechanical Analysis Framework for Rehabilitation

TL;DR

RePose addresses the need for real-time, accurate monitoring and feedback during rehabilitation by presenting an end-to-end multi-camera pipeline that estimates 3D human pose from RGB video. It combines fast single-target tracking, 2D pose estimation, learnable triangulation for 3D reconstruction, and a modified SmoothNet for real-time temporal refinement, augmented with FABRIK-based muscle-force visualization in Unity. The framework achieves approximately 30 FPS on Linux and demonstrates competitive accuracy on the Human3.6M benchmark while enabling real-time muscle-stress visualization to guide patients. This approach holds promise for remote, home-based rehabilitation by delivering immediate guidance, reducing reliance on expensive motion capture, and potentially easing clinical burdens.

Abstract

We propose a real-time 3D human pose estimation and motion analysis method termed RePose for rehabilitation training. It is capable of real-time monitoring and evaluation of patients'motion during rehabilitation, providing immediate feedback and guidance to assist patients in executing rehabilitation exercises correctly. Firstly, we introduce a unified pipeline for end-to-end real-time human pose estimation and motion analysis using RGB video input from multiple cameras which can be applied to the field of rehabilitation training. The pipeline can help to monitor and correct patients'actions, thus aiding them in regaining muscle strength and motor functions. Secondly, we propose a fast tracking method for medical rehabilitation scenarios with multiple-person interference, which requires less than 1ms for tracking for a single frame. Additionally, we modify SmoothNet for real-time posture estimation, effectively reducing pose estimation errors and restoring the patient's true motion state, making it visually smoother. Finally, we use Unity platform for real-time monitoring and evaluation of patients' motion during rehabilitation, and to display the muscle stress conditions to assist patients with their rehabilitation training.
Paper Structure (15 sections, 15 equations, 8 figures, 3 tables)

This paper contains 15 sections, 15 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Human Pose Estimation
  4. Musculoskeletal Movement foress
  5. Rehabilitation Systems
  6. Method
  7. Human Pose Estimation Model
  8. Calculate muscle forces
  9. Experiment
  10. Datasets and Evaluation Metric
  11. Implementation Details
  12. Main Results
  13. Visualization
  14. Future works
  15. Conclusion

Figures (8)

  • Figure 1: Overview of introduced RePose multi-camera real-time digital human pipeline. The red box represents the tracking box. Each camera generates a 2D pose and synthesize a 3D pose. And then generates to digital human.
  • Figure 2: Flowchart of the proposed real-time single-person tracking algorithm with hierarchical decision logic designed for complex multi-person rehabilitation scenarios.
  • Figure 3: Architecture of the modified SmoothNet for real-time temporal refinement and motion prediction.
  • Figure 4: The musculoskeletal digital human model used for muscle stress quantification and visualization.
  • Figure 5: Color-coded heatmap representation for muscle stress visualization during rehabilitation exercises.
  • ...and 3 more figures