Feelings, Not Feel: Affective Audio-Visual Pseudo-Haptics in Hand-Tracked XR

TL;DR

It is suggested that pseudo-haptics in XR may be better understood as an affective feedback channel rather than a direct replacement for physical touch in controller-free systems.

Abstract

Hand-tracking enables controller-free XR interaction but does not have the tactile feedback controllers provide. Rather than treating this solely as a missing-sensation problem, we explore whether pseudo-haptic cues on an embodied virtual hand act as tactile or as affect substitutes that shape how interactions feel. We used a mixed reality prototype that keeps the contacted surface visually neutral, rendering cues on the hand with motion modulation for texture, color glow, and movement-coupled sound. In a within-subjects study (n=12), participants experienced 12 conditions (4 effects x 3 modalities: audio, visual, both) and reported subjective affect and cognitive demand. Participants rarely reported sustained tactile, thermal sensations, yet affect shifted systematically: rough-hot lowered valence increasing arousal, while smooth-cold produced calmer pleasant states. These findings suggest that pseudo-haptics in XR may be better understood as an affective feedback channel rather than a direct replacement for physical touch in controller-free systems.

Figures (1)

• Figure 1: Affect Grid results for hand-targeted pseudo-haptic cues. Left: Mean valence–arousal ratings for each effect–modality combination. Right: Distribution of individual participant ratings, showing clustering patterns across the valence–arousal space. Color denotes effect type (rough, smooth, hot, cold), and shape indicates modality (audio, visual, audio–visual).