Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Comparative Study of Human Activity Recognition: Motion, Tactile, and multi-modal Approaches

Valerio Belcamino, Nhat Minh Dinh Le, Quan Khanh Luu, Alessandro Carfì, Van Anh Ho, Fulvio Mastrogiovanni

TL;DR

The paper addresses HAR for human-robot collaboration by comparing a vision-based tactile sensor (Taclink) with an IMU-based TER glove and proposing a multi-modal fusion framework. It implements three configurations—MBC, TBC, and MMC—using HART and ViViT backbones with late fusion, and evaluates both offline on segmented data and online on continuous sequences. Results show that MMC achieves the highest offline $F_1$ score of 95.60% and maintains strong online performance (frame-level accuracy 83.92%), highlighting the complementary strengths of motion and tactile sensing. The findings inform sensor deployment and fusion strategies for robust, real-time HAR in HRC applications.

Abstract

Human activity recognition (HAR) is essential for effective Human-Robot Collaboration (HRC), enabling robots to interpret and respond to human actions. This study evaluates the ability of a vision-based tactile sensor to classify 15 activities, comparing its performance to an IMU-based data glove. Additionally, we propose a multi-modal framework combining tactile and motion data to leverage their complementary strengths. We examined three approaches: motion-based classification (MBC) using IMU data, tactile-based classification (TBC) with single or dual video streams, and multi-modal classification (MMC) integrating both. Offline validation on segmented datasets assessed each configuration's accuracy under controlled conditions, while online validation on continuous action sequences tested online performance. Results showed the multi-modal approach consistently outperformed single-modality methods, highlighting the potential of integrating tactile and motion sensing to enhance HAR systems for collaborative robotics.

A Comparative Study of Human Activity Recognition: Motion, Tactile, and multi-modal Approaches

TL;DR

The paper addresses HAR for human-robot collaboration by comparing a vision-based tactile sensor (Taclink) with an IMU-based TER glove and proposing a multi-modal fusion framework. It implements three configurations—MBC, TBC, and MMC—using HART and ViViT backbones with late fusion, and evaluates both offline on segmented data and online on continuous sequences. Results show that MMC achieves the highest offline score of 95.60% and maintains strong online performance (frame-level accuracy 83.92%), highlighting the complementary strengths of motion and tactile sensing. The findings inform sensor deployment and fusion strategies for robust, real-time HAR in HRC applications.

Abstract

Human activity recognition (HAR) is essential for effective Human-Robot Collaboration (HRC), enabling robots to interpret and respond to human actions. This study evaluates the ability of a vision-based tactile sensor to classify 15 activities, comparing its performance to an IMU-based data glove. Additionally, we propose a multi-modal framework combining tactile and motion data to leverage their complementary strengths. We examined three approaches: motion-based classification (MBC) using IMU data, tactile-based classification (TBC) with single or dual video streams, and multi-modal classification (MMC) integrating both. Offline validation on segmented datasets assessed each configuration's accuracy under controlled conditions, while online validation on continuous action sequences tested online performance. Results showed the multi-modal approach consistently outperformed single-modality methods, highlighting the potential of integrating tactile and motion sensing to enhance HAR systems for collaborative robotics.
Paper Structure (5 sections, 4 figures, 2 tables)

This paper contains 5 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experimental Setup
  4. Results
  5. Conclusions

Figures (4)

  • Figure 1: An example of interaction with the Taclink. The user is grasping the deformable membrane while wearing the TER glove.
  • Figure 2: On the left is the IMU-based model, whose input size depends on the sensor configuration. On the right is the tactile-based model, which relies on a single video input.
  • Figure 3: On the left is the tactile-based approach, which relies on both cameras. On the right is the multi-modal solution combining tactile and motion information.
  • Figure 4: The plot shows the offline classification results for all the models considered in our comparison. The blue bars refer to the motion-based models, and the orange ones to the models that rely on tactile perception. Lastly, the multi-modal models are shown in green.