Towards Intuitive Human-Robot Interaction through Embodied Gesture-Driven Control with Woven Tactile Skins

TL;DR

This work introduces an embodied HRI framework that uses a capacitance-based woven tactile skin to enable intuitive gesture-driven control on curved robot surfaces. A 14-gesture vocabulary maps to task-space translations, rotations, and auxiliary commands, while a lightweight CNN-stem and temporal transformer recognizer delivers real-time, near-100% gesture accuracy. Validation on a 6-DOF robot during vision-free pick-and-place and pouring tasks shows substantial gains in efficiency (up to 57% faster) and high reliability, highlighting the approach's potential to narrow the gap between human intent and robot action. The study demonstrates a practical pathway toward more natural, efficient embodied HRI and points to future work on broader gesture sets, larger sensing areas, and multimodal integration.

Abstract

This paper presents a novel human-robot interaction (HRI) framework that enables intuitive gesture-driven control through a capacitance-based woven tactile skin. Unlike conventional interfaces that rely on panels or handheld devices, the woven tactile skin integrates seamlessly with curved robot surfaces, enabling embodied interaction and narrowing the gap between human intent and robot response. Its woven design combines fabric-like flexibility with structural stability and dense multi-channel sensing through the interlaced conductive threads. Building on this capability, we define a gesture-action mapping of 14 single- and multi-touch gestures that cover representative robot commands, including task-space motion and auxiliary functions. A lightweight convolution-transformer model designed for gesture recognition in real time achieves an accuracy of near-100%, outperforming prior baseline approaches. Experiments on robot arm tasks, including pick-and-place and pouring, demonstrate that our system reduces task completion time by up to 57% compared with keyboard panels and teach pendants. Overall, our proposed framework demonstrates a practical pathway toward more natural and efficient embodied HRI.

Figures (6)

• Figure 1: Framework overview: A woven tactile skin is integrated onto the robot’s surface to enable embodied, gesture-driven control (left). Gestures made on the skin are detected and translated into robot actions, facilitating intuitive human-robot interaction (right).
• Figure 2: (a) The tactile skin consists of a capacitive sensing grid formed by interlacing capacitive threads; red cotton threads are used only for channel size indication. (b-d) Applied force reduces electrode spacing and alters overlap area, producing capacitance changes.
• Figure 3: We design an intuitive gesture–action mapping that enables gesture-driven control through tactile skin attached to the robot arm's end link. 14 gestures are organized into three categories: (a–d) task-space translation ($x$, $y$, $z$ axes via push, pinch-in, and single-finger swipes), (e–g) task-space rotation ($x$, $y$, $z$ axes via two-finger swipes and circular strokes), and (h–i) auxiliary functions (five-finger pinch-in/out for initial pose and home posture).
• Figure 4: Overview of the proposed gesture recognition pipeline. From left to right: human gestures applied on the tactile skin generate spatiotemporal signals, forming the tactile input sequences. These sequences are then processed by our hybrid CNN–Transformer network to perform tactile gesture classification and produce the final prediction results.
• Figure 5: Confusion matrix comparison of our proposed hybrid architecture model (left) and the LSTM baseline (right) on the validation set. Gesture labels correspond to the definitions shown in Fig. \ref{['fig:gesture_design']}.