Social Gesture Recognition in spHRI: Leveraging Fabric-Based Tactile Sensing on Humanoid Robots

TL;DR

This work addresses the problem of enabling social touch understanding in humans-robot interaction by deploying a fabric-based, large-scale tactile skin on a humanoid robot (Reachy). It introduces a machine-knitted, resistive tactile sensor, collects a gesture dataset from 16 participants, and uses temporal features fed to a four-layer MLP to classify six gestures with an accuracy of 81.16% on held-out data. The study demonstrates the feasibility of textile skins for spHRI, highlights sensor fabrication and data collection challenges, and provides ablation findings showing the original taxel-activation feature is effective while additional features may degrade performance. The results suggest textile-based tactile sensing can enhance natural, touch-based communication in robot-assisted settings, with future work needed to broaden participant diversity and incorporate gesture meaning and context.

Abstract

Humans are able to convey different messages using only touch. Equipping robots with the ability to understand social touch adds another modality in which humans and robots can communicate. In this paper, we present a social gesture recognition system using a fabric-based, large-scale tactile sensor placed onto the arms of a humanoid robot. We built a social gesture dataset using multiple participants and extracted temporal features for classification. By collecting tactile data on a humanoid robot, our system provides insights into human-robot social touch, and displays that the use of fabric based sensors could be a potential way of advancing the development of spHRI systems for more natural and effective communication.

Figures (8)

• Figure 1: Our machine knitted large scale sensor has the ability to detect different types of gestures. Features are extracted from the sensor reading and fed into a model to classify the gesture being performed.
• Figure 2: The model pollen_robotics_documentation and actual pollen robotics arm. (A) The location that the double sided tape was placed and the holes which were filled with duct tap. (B) The sensor attached to the arm with corresponding sensing area measurements. The area where the horizontal stripes span on the upper and lower arm(the orange and purple box) are the sensing area.
• Figure 3: Reachy's arm in differing positions and the corresponding sensor reading.
• Figure 4: Display of the gestures and the signal reading. (A) A visual of the gesture being performed on the upper portion of the sensor. (B) A grid displaying what the raw signal looks like. The values shown are the mean taxel value. (C) the feature extracted from the sensor reading, the number of taxels activated per frame.
• Figure 5: (A) Indenter (yellow semi sphere) with diameter of 20.25mm used (B) Taxels used for characterization experiments.