Scene2Hap: Combining LLMs and Physical Modeling for Automatically Generating Vibrotactile Signals for Full VR Scenes

TL;DR

Scene2Hap addresses the challenge of creating scalable, immersive vibrotactile feedback for complex VR scenes by integrating a multimodal LLM-based inference pipeline with physics-aware real-time haptic rendering. It automates per-object haptic design through four chained agents that infer semantics and material properties from multimodal scene data, then converts audio signals into vibrotactile feedback. At runtime, a contact-graph-based propagation model attenuates vibrations across scene materials, producing spatially coherent haptic cues aligned with touch location. Two user studies demonstrate accurate semantic/physical inferences and improved usability, material perception, and spatial awareness when vibration propagation is considered. This hybrid approach enables real-time, adaptive haptics at scale, offering a practical path toward default, rich haptic experiences in future VR and mixed-reality environments.

Abstract

Haptic feedback contributes to immersive virtual reality (VR) experiences. Designing such feedback at scale, for all objects within a VR scene and their respective arrangements, remains a time-consuming task. We present Scene2Hap, an LLM-centered system that automatically designs object-level vibrotactile feedback for entire VR scenes based on the objects' semantic attributes and physical context. Scene2Hap employs a multimodal large language model to estimate the semantics and physical context of each object, including its material properties and vibration behavior, from the multimodal information present in the VR scene. This semantic and physical context is then used to create plausible vibrotactile signals by generating or retrieving audio signals and converting them to vibrotactile signals. For the more realistic spatial rendering of haptics in VR, Scene2Hap estimates the propagation and attenuation of vibration signals from their source across objects in the scene, considering the estimated material properties and physical context, such as the distance and contact between virtual objects. Results from two user studies confirm that Scene2Hap successfully estimates the semantics and physical context of VR scenes, and the physical modeling of vibration propagation improves usability, perceived materiality, and spatial awareness.

Figures (8)

• Figure 1: Scene2Hap is an LLM-centered system that automatically designs object-level vibrotactile feedback for the entire VR scene. It comprises two main parts: (1) LLM-Based Haptic Inference for estimating each object's semantics, material properties, and a vibrotactile signal, and (2) Physics-Aware Haptic Rendering for calculating vibrotactile feedback based on the estimated material properties and real-time spatial attributes.
• Figure 2: LLM-based haptic inference estimates the haptic properties of virtual objects by using a multi-agent workflow comprising four chained LLM agents: Scene Analyzer, Object Analyzer, Material Property Estimator, and Vibration Describer. The images show one specific example.
• Figure 3: Physics-aware haptic rendering builds the scene hierarchy and calculates all vibration propagation paths from vibration sources in real-time.
• Figure 4: In this paper, we used two development scenes to develop the prompts for our LLM-based haptic inference and four test scenes to evaluate the capability of our system. The table includes detailed information for each scene and indicates our system correctly estimated its scene category regardless of the inappropriate scene names defined by the downloaded scenes. Processing time is the time required to complete the LLM-based haptic inference for the entire scene.
• Figure 5: Participants' ratings of the correctness of LLM output on a 5-point Likert scale (1=fully incorrect -- 5=fully correct). The results show that the LLM-based haptic inference successfully infers the semantics of diverse virtual objects in alignment with human raters for most objects.