From Soft Materials to Controllers with NeuroTouch: A Neuromorphic Tactile Sensor for Real-Time Gesture Recognition

TL;DR

NeuroTouch introduces a vision-based soft-material tactile sensor that combines a deformable silicone gel with a neuromorphic DAVIS camera to achieve real-time, multi-finger gesture recognition. The pipeline fuses event-based tracking of 177 surface markers with RANSAC-inspired contact-point localization and a transformation-based classifier to identify five gesture types and estimate gesture intensity, all on CPU-only hardware. A 25-minute gesture dataset from five users demonstrates 91% gesture-type accuracy, 3.41 mm contact-point localization error, and 0.96 mm intensity error, with a runtime architecture supporting around 100 Hz inference and near-ideal low-latency performance. The work lays the groundwork for scalable, accessible, vision-based tactile interfaces in gaming, AR/VR, and assistive technologies, and provides a public dataset to spur further research in soft-material gesture sensing.

Abstract

This work presents NeuroTouch, an optical-based tactile sensor that combines a highly deformable dome-shaped soft material with an integrated neuromorphic camera, leveraging frame-based and dynamic vision for gesture detection. Our approach transforms an elastic body into a rich and nuanced interactive controller by tracking markers printed on its surface with event-based methods and harnessing their trajectories through RANSAC-based techniques. To benchmark our framework, we have created a 25 min gesture dataset, which we make publicly available to foster research in this area. Achieving over 91% accuracy in gesture classification, a 3.41 mm finger localization distance error, and a 0.96 mm gesture intensity error, our real-time, lightweight, and low-latency pipeline holds promise for applications in video games, augmented/virtual reality, and accessible devices. This research lays the groundwork for advancements in gesture detection for vision-based soft-material input technologies. Dataset: Coming Soon, Video: Coming Soon

Figures (14)

• Figure 1: APS frame (a) and EVS events (b) example of a two-finger Clockwise Twist gesture. Events are accumulated in a 10 ms frame, with positive brightness changes in green and negative ones in red.
• Figure 2: Illustration of the basic gesture types used to classify the user’s actions on the gel. Arrows indicate the direction of finger movements.
• Figure 3: Complete gesture detection framework. Our method leverages marker displacement data to localize contact points, classify gestures and estimate their intensity. Additionally, a resting position detection is performed on the APS frames.
• Figure 4: Prediction examples from the dataset. Pink (resp. blue) dots represent the predicted (resp. labeled) contact points on the events image space. Intensity is scaled by the radius of the gel (30 mm).
• Figure 5: Gesture type and contact point count classification metrics