Enhancing Physical Human-Robot Interaction: Recognizing Digits via Intrinsic Robot Tactile Sensing

TL;DR

This work demonstrates that intrinsic tactile sensing from a collaborative robot's joint torque measurements can accurately classify handwritten digits drawn on an uninstrumented touchpad, enabling intuitive pHRI without extra sensing hardware. A public dataset pHRI-DIGI-TACT combines 7‑DOF joint torques with end‑effector forces/moments, and a Bi‑LSTM classifier achieves $94\%$ online accuracy across users and conditions, including unseen participants. The study introduces data augmentation to handle reversed and rotated digits, achieving around $92\%$ and $81$–$86\%$ online accuracy for those variants, respectively, while maintaining high offline performance. A real‑world fruit delivery demonstration with an HFSM and text‑to‑speech feedback showcases the practical impact of digit‑driven pHRI for everyday tasks.

Abstract

Physical human-robot interaction (pHRI) remains a key challenge for achieving intuitive and safe interaction with robots. Current advancements often rely on external tactile sensors as interface, which increase the complexity of robotic systems. In this study, we leverage the intrinsic tactile sensing capabilities of collaborative robots to recognize digits drawn by humans on an uninstrumented touchpad mounted to the robot's flange. We propose a dataset of robot joint torque signals along with corresponding end-effector (EEF) forces and moments, captured from the robot's integrated torque sensors in each joint, as users draw handwritten digits (0-9) on the touchpad. The pHRI-DIGI-TACT dataset was collected from different users to capture natural variations in handwriting. To enhance classification robustness, we developed a data augmentation technique to account for reversed and rotated digits inputs. A Bidirectional Long Short-Term Memory (Bi-LSTM) network, leveraging the spatiotemporal nature of the data, performs online digit classification with an overall accuracy of 94\% across various test scenarios, including those involving users who did not participate in training the system. This methodology is implemented on a real robot in a fruit delivery task, demonstrating its potential to assist individuals in everyday life. Dataset and video demonstrations are available at: https://TS-Robotics.github.io/pHRI-DIGI/.

Figures (8)

• Figure 1: Conceptual overview of the proposed pHRI system: the system uses intrinsic robot torque sensing to recognize digits (0 - 9) drawn on a touchpad, including standard, reversed, and rotated variations.
• Figure 2: System architecture illustrating the data flow from the robot’s intrinsic joint torque sensing, through dataset collection and data augmentation, into the Bi-LSTM classification network. The classified patterns are then used to define the robot interface and guide the corresponding task execution.
• Figure 3: Illustration of digit drawing protocols: (Top) Standard digit drawing patterns used for data collection. (Middle) Reversed digit drawing patterns. (Bottom) Example of rotated digit drawing patterns.
• Figure 4: Mean end-effector forces and moments and standard deviations during the drawing of the digit '4' by three different users. Data represents the average touch profile for 50 trials per user, revealing consistent trends.
• Figure 5: t-SNE plot showing the clustering of the multi-user dataset (1500 data points) in a reduced two-dimensional space, colored by digit class. The distinct clusters indicate high separability between digit classes.