Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Biomechanically consistent real-time action recognition for human-robot interaction

Wanchen Li, Kahina Chalabi, Sabbah Maxime, Thomas Bousquet, Robin Passama, Sofiane Ramdani, Andrea Cherubini, Vincent Bonnet

TL;DR

This work tackles real-time online human action recognition (OHAR) in industrial contexts by leveraging biomechanically constrained joint-angle kinematics estimated from two cameras, rather than traditional joint-center positions. The authors integrate a real-time IK pipeline with a temporal-transformer that uses a causal mask and per-time-step supervision, trained with a combined cross-entropy and temporal-smoothing loss. A new HUMAR-2024 dataset with 11 subjects and 17 action labels validates viewpoint-robustness and fine-grained recognition across motion categories, achieving 88.1% accuracy in real-time. A preliminary online interaction experiment demonstrates the pipeline controlling a simulated robot, underscoring practical potential for safe, responsive human–robot collaboration in dynamic environments.

Abstract

This paper presents a novel framework for real-time human action recognition in industrial contexts, using standard 2D cameras. We introduce a complete pipeline for robust and real-time estimation of human joint kinematics, input to a temporally smoothed Transformer-based network, for action recognition. We rely on a new dataset including 11 subjects performing various actions, to evaluate our approach. Unlike most of the literature that relies on joint center positions (JCP) and is offline, ours uses biomechanical prior, eg. joint angles, for fast and robust real-time recognition. Besides, joint angles make the proposed method agnostic to sensor and subject poses as well as to anthropometric differences, and ensure robustness across environments and subjects. Our proposed learning model outperforms the best baseline model, running also in real-time, along various metrics. It achieves 88% accuracy and shows great generalization ability, for subjects not facing the cameras. Finally, we demonstrate the robustness and usefulness of our technique, through an online interaction experiment, with a simulated robot controlled in real-time via the recognized actions.

Biomechanically consistent real-time action recognition for human-robot interaction

TL;DR

This work tackles real-time online human action recognition (OHAR) in industrial contexts by leveraging biomechanically constrained joint-angle kinematics estimated from two cameras, rather than traditional joint-center positions. The authors integrate a real-time IK pipeline with a temporal-transformer that uses a causal mask and per-time-step supervision, trained with a combined cross-entropy and temporal-smoothing loss. A new HUMAR-2024 dataset with 11 subjects and 17 action labels validates viewpoint-robustness and fine-grained recognition across motion categories, achieving 88.1% accuracy in real-time. A preliminary online interaction experiment demonstrates the pipeline controlling a simulated robot, underscoring practical potential for safe, responsive human–robot collaboration in dynamic environments.

Abstract

This paper presents a novel framework for real-time human action recognition in industrial contexts, using standard 2D cameras. We introduce a complete pipeline for robust and real-time estimation of human joint kinematics, input to a temporally smoothed Transformer-based network, for action recognition. We rely on a new dataset including 11 subjects performing various actions, to evaluate our approach. Unlike most of the literature that relies on joint center positions (JCP) and is offline, ours uses biomechanical prior, eg. joint angles, for fast and robust real-time recognition. Besides, joint angles make the proposed method agnostic to sensor and subject poses as well as to anthropometric differences, and ensure robustness across environments and subjects. Our proposed learning model outperforms the best baseline model, running also in real-time, along various metrics. It achieves 88% accuracy and shows great generalization ability, for subjects not facing the cameras. Finally, we demonstrate the robustness and usefulness of our technique, through an online interaction experiment, with a simulated robot controlled in real-time via the recognized actions.
Paper Structure (14 sections, 11 equations, 4 figures, 5 tables)

This paper contains 14 sections, 11 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Methods
  4. Real-time estimation of human joint kinematics
  5. Dataset
  6. Transformer network
  7. Temporal causal mask
  8. Loss function
  9. Implementation details
  10. Evaluation metrics
  11. Results
  12. Online Human Action Recognition
  13. Experiment in simulation using OHAR for human-robot interaction
  14. Conclusion

Figures (4)

  • Figure 1: Representation of the real-time inverse kinematics pipeline (a). 22 degrees-of-freedom biomechanical model of the human body used in this study (b). For readability, only the joint angles of the right side of the body are explicated in the figure.
  • Figure 2: The Transformer takes as input joint angle sequences extracted from a sliding window, and outputs recognition result at each time step.
  • Figure 3: Eight representative tasks shown using one camera view and their corresponding inverse kinematics reconstruction.
  • Figure 4: Snapshots of the proposed prelininary experiment, described in the accompagnying video, performed with a simulated robot. t=0s: standing, robot INIT state. t=10s: sitting down, robot PEN DOWN state. t=20s: left arm picking, robot PEN RIGHT state (control in mirror). t=40s: right arm picking, robot PEN LEFT state. t=80s: two arms picking, robot PEN BACKWARD state. t=110s: stop sign, robot PEN PAUSE state