HOICraft: In-Situ VLM-based Authoring Tool for Part-Level Hand-Object Interaction Design in VR

TL;DR

HOICraft tackles the labor-intensive problem of designing part-level Hand-Object Interactions (HOI) in VR by fusing Vision–Language Models for object analysis with an in-context learning HOI mapping module. It derives a five-design HOI space from a formative study and validates an empirical data collection (Study 1) to ground the mapping in user preferences and performance. A subsequent user study (Study 2) shows HOICraft can match manual quality while significantly reducing exploratory effort and cognitive load, thanks to automated part selection, ranked recommendations with rationales, and in-situ customization. The tool enables rapid, context-aware HOI prototyping in VR and demonstrates how AI-assisted authoring can complement designers across expertise levels, with potential extensions to XR platforms and automatic preprocessing. Overall, HOICraft advances practical, scalable HOI authoring by integrating intelligent recommendations, customization, and immediate feedback within the immersive design loop.

Abstract

Hand-Object Interaction (HOI) is a key interaction component in Virtual Reality (VR). However, designing HOI still requires manual efforts to decide how object should be selected and manipulated, while also considering user abilities, which leads to time-consuming refinements. We present HOICraft, a VLM-based in-situ HOI authoring tool that enables part-level interaction design in VR. Here, HOICraft assists designers by recommending interactable elements from 3D objects, customizing HOI design properties, and mapping hand movement with virtual object behavior. We conducted a formative study with three expert VR designers to identify five representative HOI designs to support diverse user experiences. Building upon preference data from 20 participants, we develop an HOI mapping module with in-context learning. In a user study with 12 VR interaction designers, HOI mapping from HOICraft significantly reduced trial-and-error iterations compared to manual authoring. Finally, we assessed the usability of HOICraft, demonstrating its effectiveness for HOI design in VR.

Figures (14)

• Figure 1: Representative HOI design methods for virtual objects from the formative study. PM refers to physics-based manipulation. GM and GA represent gestured-based manipulation and animation accordingly. CM and CA mean contact-based manipulation and animation.
• Figure 2: Key metrics for HOI design. The metrics reflect diverse design intents of different user types and contexts.
• Figure 3: Selected 13 object–part pairs for user experience data collection in Study 1. Green bounding boxes show collider/trigger regions defined based on the mesh size of each part. Pink arrows indicate embedded motion constraints, which we predefined according to the natural behavior of each object (straight arrows for translation and curved arrows for rotation).
• Figure 4: Task 1 overview. (A) Participants compared HOI designs in pairwise trials. (B) After completing all comparisons, they reviewed and adjusted the ranking. Finally, they rated each HOI on Likert scales and explained their choices through think-aloud.
• Figure 5: Task 2 overview. (A) Participants matched object-part pairs to target states using different HOI designs (PM, GM, CM), followed by a post interview. (B) We used a microwave (handle, dial), a coffee machine (button, portafilter), a cabinet (drawer, door), and a padlock (shackle). Red-colored parts indicate the target state to be achieved.