GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality

TL;DR

GesPrompt addresses the challenge of conveying complex spatial-temporal intent to LLM-based copilots in XR by integrating co-speech gestures with speech. The proposed architecture combines an LLM system with a Gesture Processor to translate multimodal inputs into backend XR function calls, enabling intuitive object manipulation in VR. A prototype Unity-based VR system demonstrates the workflow, and a three-session user study shows improved usability, reduced cognitive load, and faster interaction compared with gesture-only and voice-only baselines. The approach has potential to generalize to AR and desktop settings, paving the way for richer, more natural multimodal interactions in XR copilots and content creation.

Abstract

Large Language Model (LLM)-based copilots have shown great potential in Extended Reality (XR) applications. However, the user faces challenges when describing the 3D environments to the copilots due to the complexity of conveying spatial-temporal information through text or speech alone. To address this, we introduce GesPrompt, a multimodal XR interface that combines co-speech gestures with speech, allowing end-users to communicate more naturally and accurately with LLM-based copilots in XR environments. By incorporating gestures, GesPrompt extracts spatial-temporal reference from co-speech gestures, reducing the need for precise textual prompts and minimizing cognitive load for end-users. Our contributions include (1) a workflow to integrate gesture and speech input in the XR environment, (2) a prototype VR system that implements the workflow, and (3) a user study demonstrating its effectiveness in improving user communication in VR environments.

Figures (15)

• Figure 1: An example usage of GesPrompt: (a) An end-user wants to hang a virtual painting on the wall using GesPrompt: (a-1) The user asks the system to select the "Starry Night" painting from a pile of paintings on the floor by gesturing the rough location of the painting; (a-2) in the same utterance, the user also specifies the location of the painting using gesture; (a-3) realizing that the painting is not level, the user instructs the system to adjust its alignment while gesturing the appropriate amount of rotation; (b) The user stands back and sees that the painting is too small, then asks the system to enlarge the painting:(b-1) (b-2) the user sets the size of the painting by gesturing; (b-3) the user is satisfied with the painting, now properly placed and resized to fit the room's aesthetics. (c) Then the user is excited and wants to take a virtual video with the painting: (c-1) The user asks the system to set up the path of the camera by gesturing; (c-2) they also want the camera to look at a certain direction, which is pointed out by the user; (c-3) the camera moves along the path and records the video as the user desires.
• Figure 2: System Workflow: The system modifies the XR environment using speech and gesture prompts, processed by two core components: the LLM system and the Gesture Processor. The LLM system converts speech to text with timestamps, identifies operation functions and parameters, and extracts unambiguous values from the XR scene. The Gesture Processor resolves ambiguous parameters by analyzing gestures segmented using speech timestamps and extracting values from the XR scene. Once all parameters are identified, the system updates the XR environment. If the system fails to update the environment, an error message will be sent to the user.
• Figure 3: Example showcase of deictic gestures and the spatial parameters. For the Position parameter, the gesture could be pointing toward a specific location. For the Object parameter, the gesture could be pointing directly at an object. For the Direction parameter, the gesture could be a directional pointing motion to indicate the desired orientation
• Figure 4: Examples of iconic gestures and the corresponding spatial parameters. For the Rotation parameter, gestures could include the rotation of one (a) or both (b) hands to represent rotational movements. For the Size parameter, gestures could indicate the relative size of an object using the distance between fingers (c), hands (d), or between a hand and a reference entity (e). For the Path parameter, gestures could involve circular hand motions (f) outlining the shape of a sphere or demonstrating the trajectory of an object.
• Figure 5: LLM System consists of a speech-to-text (STT) module and a GPT module. It processes user voice input and data from the XR environment. It then outputs timestamps, parameter values, and tokens to the gesture processor and the XR environment for further processing.