NavAI: A Generalizable LLM Framework for Navigation Tasks in Virtual Reality Environments
Xue Qin, Matthew DiGiovanni
TL;DR
NavAI tackles robust navigation in immersive VR by leveraging an LLM-driven pipeline that interprets scenes and maps user intents to VR controls. The framework combines a Comprehensive Interpreter, Pre-defined Navigation Categories, a multi-LLM Decision Voter, and a Decision-to-Control Mapping to support both basic actions and goal-directed tasks across three Unity VR environments. It achieves an 89% success rate on direct goal-oriented tasks and demonstrates strong Action Navigator performance, though interpretation and decision-making latency limit real-time applicability. The work highlights practical paths forward, including edge-based, lightweight local models and parallel inference to reduce latency and enable scalable, real-time VR navigation.
Abstract
Navigation is one of the fundamental tasks for automated exploration in Virtual Reality (VR). Existing technologies primarily focus on path optimization in 360-degree image datasets and 3D simulators, which cannot be directly applied to immersive VR environments. To address this gap, we present NavAI, a generalizable large language model (LLM)-based navigation framework that supports both basic actions and complex goal-directed tasks across diverse VR applications. We evaluate NavAI in three distinct VR environments through goal-oriented and exploratory tasks. Results show that it achieves high accuracy, with an 89% success rate in goal-oriented tasks. Our analysis also highlights current limitations of relying entirely on LLMs, particularly in scenarios that require dynamic goal assessment. Finally, we discuss the limitations observed during the experiments and offer insights for future research directions.