Teaching LLMs to See and Guide: Context-Aware Real-Time Assistance in Augmented Reality

TL;DR

The paper tackles real-time, context-aware AR/VR task guidance by integrating multimodal streams into a large language model (LLM) assistant. It introduces an incremental prompting framework with four models, ranging from minimal context to full multimodal context, to quantify how contextual richness affects performance. Evaluation on the HoloAssist dataset uses lexical metrics, an LLM-based judge, and human evaluators, complemented by All-Except-One and Only-One ablations. Results show that richer context, especially hand actions and current task steps, yields more accurate and relevant guidance and that LLM judgments align closely with human assessments. The work demonstrates the potential of multimodal LLMs for adaptive, real-time AR/VR instruction and outlines future directions for proactive guidance and expanded modalities.

Abstract

The growing adoption of augmented and virtual reality (AR and VR) technologies in industrial training and on-the-job assistance has created new opportunities for intelligent, context-aware support systems. As workers perform complex tasks guided by AR and VR, these devices capture rich streams of multimodal data, including gaze, hand actions, and task progression, that can reveal user intent and task state in real time. Leveraging this information effectively remains a major challenge. In this work, we present a context-aware large language model (LLM) assistant that integrates diverse data modalities, such as hand actions, task steps, and dialogue history, into a unified framework for real-time question answering. To systematically study how context influences performance, we introduce an incremental prompting framework, where each model version receives progressively richer contextual inputs. Using the HoloAssist dataset, which records AR-guided task executions, we evaluate how each modality contributes to the assistant's effectiveness. Our experiments show that incorporating multimodal context significantly improves the accuracy and relevance of responses. These findings highlight the potential of LLM-driven multimodal integration to enable adaptive, intuitive assistance for AR and VR-based industrial training and assistance.

Figures (14)

• Figure 1: Illustration of our incremental prompting methodology. Contextual datasets, restricted to real-time information up to time $T$, are progressively provided to multiple models. Each model receives increasingly detailed context, starting from basic task duration information to detailed hand actions data. The models generate responses based on this incremental context, and these outputs undergo a multi-faceted evaluation, including lexical comparison, LLM-as-a-judge, and human evaluation.
• Figure 2: Average final score (mean ± standard deviation) across four models as evaluated by the proposed LLM-as-a-judge. Performance improves consistently as additional contextual signals are introduced, with Model 4 achieving the highest accuracy and consistency.
• Figure 3: LLM-as-a-judge's dimension-wise evaluation scores for each model, showing trends across correctness, completeness, contextual relevance, and clarity. All dimensions exhibit consistent improvement as contextual components are added, with Model 4 achieving the highest average scores in all categories.
• Figure 4: Winner share percentage across models based on LLM-as-a-judge evaluations. Model 4 dominates with $69.1\%$ of top-ranked responses, indicating that contextual enrichment strongly enhances perceived response quality.
• Figure 5: Robustness analysis of the LLM-as-a-judge framework using different judge models (GPT-4o, GPT-4.1, and GPT-5). All exhibit a consistent upward trend, demonstrating that the framework’s evaluations are stable across different models.