Vibro-Sense: Robust Vibration-based Impulse Response Localization and Trajectory Tracking for Robotic Hands

TL;DR

Vibro-Sense demonstrates that simple structure-borne vibrations captured by a sparse microphone array on a robotic hand can enable high-precision touch localization and dynamic trajectory tracking. By employing seven piezoelectric microphones and an Audio Spectrogram Transformer, the approach maps vibro-signals to contact locations and hand trajectories, achieving static localization errors under 5 mm and revealing material-dependent cues that distinguish impulse localization from friction-driven tracking. A key insight is that stiff materials favor sharp impulse signatures, while textured materials provide rich frictional cues for continuous tracking, with robustness to the robot’s own motion. The work provides open-source datasets, models, and experimental setups to spur accessible, scalable contact perception for manipulation.

Abstract

Rich contact perception is crucial for robotic manipulation, yet traditional tactile skins remain expensive and complex to integrate. This paper presents a scalable alternative: high-accuracy whole-body touch localization via vibro-acoustic sensing. By equipping a robotic hand with seven low-cost piezoelectric microphones and leveraging an Audio Spectrogram Transformer, we decode the vibrational signatures generated during physical interaction. Extensive evaluation across stationary and dynamic tasks reveals a localization error of under 5 mm in static conditions. Furthermore, our analysis highlights the distinct influence of material properties: stiff materials (e.g., metal) excel in impulse response localization due to sharp, high-bandwidth responses, whereas textured materials (e.g., wood) provide superior friction-based features for trajectory tracking. The system demonstrates robustness to the robot's own motion, maintaining effective tracking even during active operation. Our primary contribution is demonstrating that complex physical contact dynamics can be effectively decoded from simple vibrational signals, offering a viable pathway to widespread, affordable contact perception in robotics. To accelerate research, we provide our full datasets, models, and experimental setups as open-source resources.

Figures (7)

• Figure 1: Setup of the impulse response localization task. The UR5e robotic arm is shown applying controlled pokes to the hand using a solenoid actuator with a metal indenter.
• Figure 2: Schematic diagram of the microphones' localization, represented by the red area. The microphones on the RH8D hand are mounted externally.
• Figure 3: Magnitude spectra (in dB) of all seven channels. Each spectrum is normalized such that the maximum magnitude is 0 dB. Frequency is shown in kHz.
• Figure 4: Test distance by frequency and window size ($n_{\text{fft}}$). Mean distances (mm) with 10 repetitions, showing the influence of frequency and spectral resolution.
• Figure 5: Forearm localization error: half violin plots showing the distribution of prediction error (Euclidean Distance in mm) for four materials on the forearm section, broken down by view.