Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HapticVLM: VLM-Driven Texture Recognition Aimed at Intelligent Haptic Interaction

Muhammad Haris Khan, Miguel Altamirano Cabrera, Dmitrii Iarchuk, Yara Mahmoud, Daria Trinitatova, Issatay Tokmurziyev, Dzmitry Tsetserukou

TL;DR

HapticVLM addresses the challenge of sensor-free, context-aware haptic rendering by fusing visual-material recognition with ambient temperature inference. The system combines a ConvNeXt-based material encoder with a Vision-Language Model to drive vibrotactile feedback and Peltier-based thermal cues, enabling real-time multisensory texture perception. Experimental results show an average haptic-pattern recognition accuracy of 84.7% and temperature estimation accuracy of 86.7% within a broad tolerance, highlighting the potential for immersive VR, teleoperation, and assistive applications. Limitations include a small pattern set and participant pool, with future work aimed at expanding tactile patterns and user studies to validate and extend the approach.

Abstract

This paper introduces HapticVLM, a novel multimodal system that integrates vision-language reasoning with deep convolutional networks to enable real-time haptic feedback. HapticVLM leverages a ConvNeXt-based material recognition module to generate robust visual embeddings for accurate identification of object materials, while a state-of-the-art Vision-Language Model (Qwen2-VL-2B-Instruct) infers ambient temperature from environmental cues. The system synthesizes tactile sensations by delivering vibrotactile feedback through speakers and thermal cues via a Peltier module, thereby bridging the gap between visual perception and tactile experience. Experimental evaluations demonstrate an average recognition accuracy of 84.67% across five distinct auditory-tactile patterns and a temperature estimation accuracy of 86.7% based on a tolerance-based evaluation method with an 8°C margin of error across 15 scenarios. Although promising, the current study is limited by the use of a small set of prominent patterns and a modest participant pool. Future work will focus on expanding the range of tactile patterns and increasing user studies to further refine and validate the system's performance. Overall, HapticVLM presents a significant step toward context-aware, multimodal haptic interaction with potential applications in virtual reality, and assistive technologies.

HapticVLM: VLM-Driven Texture Recognition Aimed at Intelligent Haptic Interaction

TL;DR

HapticVLM addresses the challenge of sensor-free, context-aware haptic rendering by fusing visual-material recognition with ambient temperature inference. The system combines a ConvNeXt-based material encoder with a Vision-Language Model to drive vibrotactile feedback and Peltier-based thermal cues, enabling real-time multisensory texture perception. Experimental results show an average haptic-pattern recognition accuracy of 84.7% and temperature estimation accuracy of 86.7% within a broad tolerance, highlighting the potential for immersive VR, teleoperation, and assistive applications. Limitations include a small pattern set and participant pool, with future work aimed at expanding tactile patterns and user studies to validate and extend the approach.

Abstract

This paper introduces HapticVLM, a novel multimodal system that integrates vision-language reasoning with deep convolutional networks to enable real-time haptic feedback. HapticVLM leverages a ConvNeXt-based material recognition module to generate robust visual embeddings for accurate identification of object materials, while a state-of-the-art Vision-Language Model (Qwen2-VL-2B-Instruct) infers ambient temperature from environmental cues. The system synthesizes tactile sensations by delivering vibrotactile feedback through speakers and thermal cues via a Peltier module, thereby bridging the gap between visual perception and tactile experience. Experimental evaluations demonstrate an average recognition accuracy of 84.67% across five distinct auditory-tactile patterns and a temperature estimation accuracy of 86.7% based on a tolerance-based evaluation method with an 8°C margin of error across 15 scenarios. Although promising, the current study is limited by the use of a small set of prominent patterns and a modest participant pool. Future work will focus on expanding the range of tactile patterns and increasing user studies to further refine and validate the system's performance. Overall, HapticVLM presents a significant step toward context-aware, multimodal haptic interaction with potential applications in virtual reality, and assistive technologies.
Paper Structure (12 sections, 1 equation, 4 figures, 1 table)

This paper contains 12 sections, 1 equation, 4 figures, 1 table.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. System Architecture
  4. Material Recognition
  5. VLM-based Temperature Inference
  6. Haptic Feedback Module
  7. Experimental Evaluation
  8. VLM-driven Temperature Estimation
  9. Haptic Pattern Recognition Study
  10. Experimental Setup
  11. Results
  12. Conclusions and Future Work

Figures (4)

  • Figure 1: System overview of HapticVLM.
  • Figure 2: Material recognition via ConvNeXT and temperature estimation via VLM. (a) The first step in material recognition involves preprocessing, which utilizes an image encoder to create embeddings of known materials stored in a database. The second step inputs images from the targeted scene into the same encoder to generate descriptors, followed by calculating cosine distance to identify the material. Once confirmed, the material name serves as a key to retrieve corresponding audio feedback from another database, which is then delivered to the user via a speaker. (b) Temperature estimation is performed using a Vision-Language Model (VLM), which considers visual cues such as clothing and lighting to estimate temperature, providing feedback to users through a Peltier device.
  • Figure 3: Haptic feedback patterns (a) Metal Whooshing (MW), (b) Fabric Rubbing (FR), (c) Wood striking (WS), (d) Glass tapping (GT), and (e) Wood carving (WC)
  • Figure 4: Participant seated at a desk during the evaluation, with their right hand placed on the device for haptic feedback assessment.