HapticMatch: An Exploration for Generative Material Haptic Simulation and Interaction
Mingxin Zhang, Yu Yao, Yasutoshi Makino, Hiroyuki Shinoda, Masashi Sugiyama
TL;DR
HapticMatch tackles the bottleneck of creating high-fidelity tactile textures by enabling a Scan-to-Touch workflow that converts RGB images into renderable height maps and friction cues using conditional generative models. A new dataset of 100 materials with precisely aligned visual-haptic pairs (RGB images, GelSight height maps, and vibration audio) supports training Flow Matching and Latent Diffusion models to synthesize tactile textures from visuals. Through both qualitative visual assessments and quantitative metrics (LPIPS and PSD), diffusion-based and flow-matching approaches demonstrate superior perceptual fidelity over baselines, enabling rapid, hardware-agnostic haptic prototyping. The work also discusses rendering modalities (electrostatic and ultrasonic) and highlights potential for design acceleration and accessibility, while outlining future directions for scaling data, multi-modal fusion, and real-time engine integration.
Abstract
High-fidelity haptic feedback is essential for immersive virtual environments, yet authoring realistic tactile textures remains a significant bottleneck for designers. We introduce HapticMatch, a visual-to-tactile generation framework designed to democratize haptic content creation. We present a novel dataset containing precisely aligned pairs of micro-scale optical images, surface height maps, and friction-induced vibrations for 100 diverse materials. Leveraging this data, we explore and demonstrate that conditional generative models like diffusion and flow-matching can synthesize high-fidelity, renderable surface geometries directly from standard RGB photos. By enabling a "Scan-to-Touch" workflow, HapticMatch allows interaction designers to rapidly prototype multimodal surface sensations without specialized recording equipment, bridging the gap between visual and tactile immersion in VR/AR interfaces.