Can a Machine Feel Vibrations?: A Framework for Vibrotactile Sensation and Emotion Prediction via a Neural Network

TL;DR

This work tackles the challenge of predicting how vibrotactile Tactons will be perceived in terms of roughness and emotion (valence, arousal) without iterative user testing. It introduces a biomimetic framework comprising haptic data augmentation, mechanoreceptive processing, and VibNet, a dual-stream neural network that ingests both 1D accelerations and two-channel spectrograms derived from mechanoreceptor-inspired filters. Trained on 154 Tactons from 36 participants and evaluated on two unseen test sets, VibNet achieves superior predictive accuracy, with about 82% of predictions falling within the ground-truth standard deviation and particularly strong performance for valence. The approach enables faster Tacton prototyping and informs future multimodal haptic design, while outlining practical limitations and avenues for extending the framework to broader design spaces and modalities.

Abstract

Vibrotactile signals offer new possibilities for conveying sensations and emotions in various applications. Yet, designing vibrotactile tactile icons (i.e., Tactons) to evoke specific feelings often requires a trial-and-error process and user studies. To support haptic design, we propose a framework for predicting sensory and emotional ratings from vibration signals. We created 154 Tactons and conducted a study to collect acceleration data from smartphones and roughness, valence, and arousal user ratings (n=36). We converted the Tacton signals into two-channel spectrograms reflecting the spectral sensitivities of mechanoreceptors, then input them into VibNet, our dual-stream neural network. The first stream captures sequential features using recurrent networks, while the second captures temporal-spectral features using 2D convolutional networks. VibNet outperformed baseline models, with 82% of its predictions falling within the standard deviations of ground truth user ratings for two new Tacton sets. We discuss the efficacy of our mechanoreceptive processing and dual-stream neural network and present future research directions.

Figures (6)

• Figure 1: An overview diagram illustrating the Tacton design, user study to construct a haptic dataset, and our computational framework.
• Figure 2: Examples of 154 Tactons created using three design approaches: (a) 24 examples from 54 Tactons with varying sinusoidal parameters, (b) 24 examples from 60 Tactons with varying rhythmic structure, carrier frequency, and amplitude, and (c) 24 examples from 40 Tactons created based on VibViz seifi2015vibviz. The x-axis shows time (seconds), and the y-axis shows amplitude (-1 to 1) on iPhones.
• Figure 3: Study setup: (a) An overview of the experimental setup and (b) A screenshot of the GUI application used to collect user responses in the user study (left: training session, right: main session).
• Figure 4: Accelerations and sensory and emotional ratings of 154 Tactons collected from 36 participants. (a) Exemplar accelerations of three Tactons (V6, V100, and V122) from the three devices (iPhone 13 mini, iPhone 14, and iPhone 11 Pro Max). (b) The average ratings and standard deviations for roughness, valence, and arousal of 154 Tactons.
• Figure 5: Visualization of the three proposed augmentation techniques for mechanical vibrations, using V24 created with a carrier frequency of 155 Hz. The rows, respectively, display the acceleration waveform over the entire duration, a zoomed-in view of the waveform between 0.5 and 0.6 seconds in the time domain, and the corresponding frequency domain plot using the Fast Fourier Transform (FFT).