Large Language Model-assisted Speech and Pointing Benefits Multiple 3D Object Selection in Virtual Reality

TL;DR

The introduced technique, AssistVR, outperforms the baseline technique when there are multiple target objects, and contrary to the common belief for speech interfaces, AssistVR was able to outperform the baseline even when the target objects were difficult to reference verbally.

Abstract

Selection of occluded objects is a challenging problem in virtual reality, even more so if multiple objects are involved. With the advent of new artificial intelligence technologies, we explore the possibility of leveraging large language models to assist multi-object selection tasks in virtual reality via a multimodal speech and raycast interaction technique. We validate the findings in a comparative user study (n=24), where participants selected target objects in a virtual reality scene with different levels of scene perplexity. The performance metrics and user experience metrics are compared against a mini-map based occluded object selection technique that serves as the baseline. Results indicate that the introduced technique, AssistVR, outperforms the baseline technique when there are multiple target objects. Contrary to the common belief for speech interfaces, AssistVR was able to outperform the baseline even when the target objects were difficult to reference verbally. This work demonstrates the viability and interaction potential of an intelligent multimodal interactive system powered by large laguage models. Based on the results, we discuss the implications for design of future intelligent multimodal interactive systems in immersive environments.

Figures (8)

• Figure 1: Workflow of the speech-based object selection component of AssistVR using Azure CLU. The workflow consists of the training phase (top row) and the deployment phase (bottom row), where a model is initially trained and then deployed to process user speech input in real-time.
• Figure 2: Draggable Panel of AssistVR which displays the recognized speech, together with a list of the names and 3D previews of all selected objects in the current scene. In this example, the user states 'Select all purple spheres', and all four purple spheres in the scene are selected.
• Figure 3: Illustration of the concept of scene perplexity applied in the object selection user study. From left to right: Low/Medium/High Scene Perplexity. The left image shows the user's view after starting the search trial under the 1Target condition. The middle image shows the view of the search trial under the 2Targets condition. The right image shows the view of the repeat trial under the 4Targets condition.
• Figure 4: Bar plot of average trial completion time of AssistVR and DiscPIMmaslych2023toward in the search and repeat task. 95% confidence intervals of the mean estimates are shown.
• Figure 5: Trial completion time (seconds) for each technique across search and repeat trial types for each NumTargets condition, with 95% confidence intervals of the mean estimate.