Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Advancements in Tactile Hand Gesture Recognition for Enhanced Human-Machine Interaction

Chiara Fumelli, Anirvan Dutta, Mohsen Kaboli

TL;DR

This study tackles tactile hand-gesture recognition for a large-area textile tactile interface to enable intuitive human–machine interaction. It systematically compares traditional feature-engineering methods with deep-learning architectures (CNN, RNN/LSTM, CNN-LSTM) using a standardized data-processing framework and a shift-invariant data-augmentation strategy. Offline results show high accuracies (up to $0.98$ for feature methods and $0.97$ for DL models) with augmentation, while online evaluation reveals CNN-LSTM as the most robust real-time approach, achieving about $0.93$ accuracy despite distribution shifts. Overall, the work demonstrates the viability of textile-based tactile interfaces for practical HMI and informs hardware–software design choices to achieve robust tactile gesture recognition.

Abstract

Motivated by the growing interest in enhancing intuitive physical Human-Machine Interaction (HRI/HVI), this study aims to propose a robust tactile hand gesture recognition system. We performed a comprehensive evaluation of different hand gesture recognition approaches for a large area tactile sensing interface (touch interface) constructed from conductive textiles. Our evaluation encompassed traditional feature engineering methods, as well as contemporary deep learning techniques capable of real-time interpretation of a range of hand gestures, accommodating variations in hand sizes, movement velocities, applied pressure levels, and interaction points. Our extensive analysis of the various methods makes a significant contribution to tactile-based gesture recognition in the field of human-machine interaction.

Advancements in Tactile Hand Gesture Recognition for Enhanced Human-Machine Interaction

TL;DR

This study tackles tactile hand-gesture recognition for a large-area textile tactile interface to enable intuitive human–machine interaction. It systematically compares traditional feature-engineering methods with deep-learning architectures (CNN, RNN/LSTM, CNN-LSTM) using a standardized data-processing framework and a shift-invariant data-augmentation strategy. Offline results show high accuracies (up to for feature methods and for DL models) with augmentation, while online evaluation reveals CNN-LSTM as the most robust real-time approach, achieving about accuracy despite distribution shifts. Overall, the work demonstrates the viability of textile-based tactile interfaces for practical HMI and informs hardware–software design choices to achieve robust tactile gesture recognition.

Abstract

Motivated by the growing interest in enhancing intuitive physical Human-Machine Interaction (HRI/HVI), this study aims to propose a robust tactile hand gesture recognition system. We performed a comprehensive evaluation of different hand gesture recognition approaches for a large area tactile sensing interface (touch interface) constructed from conductive textiles. Our evaluation encompassed traditional feature engineering methods, as well as contemporary deep learning techniques capable of real-time interpretation of a range of hand gestures, accommodating variations in hand sizes, movement velocities, applied pressure levels, and interaction points. Our extensive analysis of the various methods makes a significant contribution to tactile-based gesture recognition in the field of human-machine interaction.
Paper Structure (17 sections, 5 figures, 4 tables, 1 algorithm)

This paper contains 17 sections, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Methodologies
  4. Hardware: Textile-based Tactile Interface
  5. Hand Gesture Protocols
  6. Data Augmentation
  7. Hand Gesture Recognition Methods
  8. Spatio-temporal Features Approach
  9. Touch Pattern Recognition Approach
  10. Deep Learning (DL) Approach- CNN
  11. DL Approach- Recurrent Neural Network
  12. DL Approach- Convolutional Neural Network + Long Short Term Memory
  13. Results and Discussion
  14. Offline Evaluation
  15. Online evaluation
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Example of the gesture "double tap" being performed on the touch interface TeXYZ. The bottom graph represents the evolution through time of the pressure values of each taxel (each one represented with a different color). Each matrix of the top graphs represents a snapshot of the pressure values on the sensor
  • Figure 2: The 10 Hand Gesture movements associated with the respective class
  • Figure 3: Hand Gestures Raw Signal from the TexYZ, summed over the whole time series length
  • Figure 4: Illustration of the hand gesture recognition methods evaluated in this work.
  • Figure 5: Confusion matrices of the best performing configuration for each method in the offline evaluation