HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioning

TL;DR

HapticLLaMA introduces the first multimodal sensory language model for haptic captioning, translating vibration signals into category-specific natural language descriptions. It leverages two haptic tokenizers (Frequency-based and EnCodec-based), a LoRA-enhanced LLaMA backbone, and a two-stage training pipeline augmented with RLHF via Direct Preference Optimization, trained on HapticCap and VibRate data. The approach yields strong perceptual alignment, with automated metrics showing BLEU-4 up to 32.06 and METEOR up to 59.98, and human evaluations indicating over 61% of captions rated above 3.5 on a 7-point scale, improved further by RLHF. This work demonstrates that large language models can effectively process tactile signals, enabling haptic captioning for VR, accessibility, and rehabilitation applications, and provides two tokenization strategies and a dedicated vibrotactile dataset for future research.

Abstract

Haptic captioning is the task of generating natural language descriptions from haptic signals, such as vibrations, for use in virtual reality, accessibility, and rehabilitation applications. While previous multimodal research has focused primarily on vision and audio, haptic signals for the sense of touch remain underexplored. To address this gap, we formalize the haptic captioning task and propose HapticLLaMA, a multimodal sensory language model that interprets vibration signals into descriptions in a given sensory, emotional, or associative category. We investigate two types of haptic tokenizers, a frequency-based tokenizer and an EnCodec-based tokenizer, that convert haptic signals into sequences of discrete units, enabling their integration with the LLaMA model. HapticLLaMA is trained in two stages: (1) supervised fine-tuning using the LLaMA architecture with LoRA-based adaptation, and (2) fine-tuning via reinforcement learning from human feedback (RLHF). We assess HapticLLaMA's captioning performance using both automated n-gram metrics and human evaluation. HapticLLaMA demonstrates strong capability in interpreting haptic vibration signals, achieving a METEOR score of 59.98 and a BLEU-4 score of 32.06 respectively. Additionally, over 61% of the generated captions received human ratings above 3.5 on a 7-point scale, with RLHF yielding a 10% improvement in the overall rating distribution, indicating stronger alignment with human haptic perception. These findings highlight the potential of large language models to process and adapt to sensory data.

Figures (8)

• Figure 1: HapticLLaMA can generate sensory, emotional, and associative captions for an input vibration haptic signal with a tailored prompt template.
• Figure 2: Overview of the two-stage process for constructing HapticLLaMA: (1) supervised fine-tuning, and (2) reinforcement learning from human feedback (RLHF).
• Figure 3: Two haptic tokenizers to encode vibration signals into input representations suitable for LLaMA.
• Figure 4: The distribution of human ratings after Stage 1 and 2 training for captions generated by EnCodec-based HapticLLaMA. Higher ratings are shown in blue and lower ratings are in gray.
• Figure 5: EnCodec-based HapticLLaMA's performance across sensory, emotional, and associative categories.