Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sound2Hap: Learning Audio-to-Vibrotactile Haptic Generation from Human Ratings

Yinan Li, Hasti Seifi

TL;DR

Sound2Hap tackles the poor generalization of traditional audio-to-haptic mappings by learning a perceptually aligned translation from diverse environmental sounds. It combines a data-driven dataset of 4,000 audio-vibration pairs with two autoencoder-based model variants trained on human-rated signals, achieving superior perceptual match and HX I scores compared to baselines. The work provides an open-source dataset, two trained models, and an interactive web tool to enable researchers and designers to explore, generate, and deploy sound-driven vibrotactile feedback. This approach expands the reach of sound-driven haptics for XR, accessibility, and multimodal experiences, while outlining practical limitations and directions for future real-time, multi-source, and cross-location haptic systems.

Abstract

Environmental sounds like footsteps, keyboard typing, or dog barking carry rich information and emotional context, making them valuable for designing haptics in user applications. Existing audio-to-vibration methods, however, rely on signal-processing rules tuned for music or games and often fail to generalize across diverse sounds. To address this, we first investigated user perception of four existing audio-to-haptic algorithms, then created a data-driven model for environmental sounds. In Study 1, 34 participants rated vibrations generated by the four algorithms for 1,000 sounds, revealing no consistent algorithm preferences. Using this dataset, we trained Sound2Hap, a CNN-based autoencoder, to generate perceptually meaningful vibrations from diverse sounds with low latency. In Study 2, 15 participants rated its output higher than signal-processing baselines on both audio-vibration match and Haptic Experience Index (HXI), finding it more harmonious with diverse sounds. This work demonstrates a perceptually validated approach to audio-haptic translation, broadening the reach of sound-driven haptics.

Sound2Hap: Learning Audio-to-Vibrotactile Haptic Generation from Human Ratings

TL;DR

Sound2Hap tackles the poor generalization of traditional audio-to-haptic mappings by learning a perceptually aligned translation from diverse environmental sounds. It combines a data-driven dataset of 4,000 audio-vibration pairs with two autoencoder-based model variants trained on human-rated signals, achieving superior perceptual match and HX I scores compared to baselines. The work provides an open-source dataset, two trained models, and an interactive web tool to enable researchers and designers to explore, generate, and deploy sound-driven vibrotactile feedback. This approach expands the reach of sound-driven haptics for XR, accessibility, and multimodal experiences, while outlining practical limitations and directions for future real-time, multi-source, and cross-location haptic systems.

Abstract

Environmental sounds like footsteps, keyboard typing, or dog barking carry rich information and emotional context, making them valuable for designing haptics in user applications. Existing audio-to-vibration methods, however, rely on signal-processing rules tuned for music or games and often fail to generalize across diverse sounds. To address this, we first investigated user perception of four existing audio-to-haptic algorithms, then created a data-driven model for environmental sounds. In Study 1, 34 participants rated vibrations generated by the four algorithms for 1,000 sounds, revealing no consistent algorithm preferences. Using this dataset, we trained Sound2Hap, a CNN-based autoencoder, to generate perceptually meaningful vibrations from diverse sounds with low latency. In Study 2, 15 participants rated its output higher than signal-processing baselines on both audio-vibration match and Haptic Experience Index (HXI), finding it more harmonious with diverse sounds. This work demonstrates a perceptually validated approach to audio-haptic translation, broadening the reach of sound-driven haptics.
Paper Structure (60 sections, 9 figures, 2 tables)

This paper contains 60 sections, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Signal-Processing Methods
  4. Learning-Based Methods
  5. Overview of Sound2Hap Design and Evaluation Process
  6. Creating An Audio-Vibration Dataset
  7. Audio Dataset
  8. Four Signal-Processing Methods for Audio-to-Vibration Conversion
  9. Perception-Level Mapping
  10. Frequency Shifting
  11. Pitch Matching
  12. HapticGen
  13. Normalization Process for Vibration Signals
  14. User Study 1: Collecting User Ratings for Audio-to-Vibration Techniques
  15. Study Methods
  16. ...and 45 more sections

Figures (9)

  • Figure 1: Our overall process for designing and evaluating Sound2Hap.
  • Figure 2: Examples of original audio signals (orange) and converted vibrations (blue), using four signal-processing algorithms. The audio was recorded at 44.1 kHz, 16-bit resolution, and vibrations were rendered at 8 kHz, 16-bit resolution. The y-axis of time-domain plots (left) ranges from -1 to 1. The y-axis of frequency-domain plots (right) shows magnitude, with taller spikes indicating greater energy.
  • Figure 3: Study setup and interface for the User Study 1. Participants held a voice-coil vibration actuator (Haptuator Redesign) and rated each vibration's match to the sound clips on the interface.
  • Figure 4: Performance of the four audio-to-vibration algorithms across five overall categories and 50 sound classes.
  • Figure 5: Sound2Hap model with two training schemes and loss functions: (a) Top-Pair Sound2Hap is trained to directly replicate the best-rated vibration signal; (b) Preference-Weighted Sound2Hap employs a two-phase training strategy: an initial pre-training phase using a blended vibration target, followed by an adversarial fine-tuning phase with a discriminator to refine the generator's output.
  • ...and 4 more figures