YETI (YET to Intervene) Proactive Interventions by Multimodal AI Agents in Augmented Reality Tasks

TL;DR

The paper tackles the lack of proactive, context-aware AI assistance in augmented reality by introducing YETI, a lightweight, on-device proactive intervention framework that uses multimodal signals derived from frame-level object counts and structural similarity. By combining frame sampling at $1$ FPS, a lightweight VLM (PaliGemma-3b-mix-448), and an intervention algorithm with tunable hyperparameters ($\tau$, $m$, $r$, $k$), YETI achieves real-time proactive interventions with substantially lower memory requirements than prior baselines. Empirical results on the HoloAssist benchmark show that YETI improves recall and F-measure across intervention types, while remaining computationally efficient enough for AR devices. The work also includes extensive ablations and comparisons with alternative classifiers and implicit interaction settings, and outlines future directions to integrate richer sensing modalities and broader VLM evaluations for enhanced proactive AR assistance.

Abstract

Multimodal AI Agents are AI models that have the capability of interactively and cooperatively assisting human users to solve day-to-day tasks. Augmented Reality (AR) head worn devices can uniquely improve the user experience of solving procedural day-to-day tasks by providing egocentric multimodal (audio and video) observational capabilities to AI Agents. Such AR capabilities can help AI Agents see and listen to actions that users take which can relate to multimodal capabilities of human users. Existing AI Agents, either Large Language Models (LLMs) or Multimodal Vision-Language Models (VLMs) are reactive in nature, which means that models cannot take an action without reading or listening to the human user's prompts. Proactivity of AI Agents on the other hand can help the human user detect and correct any mistakes in agent observed tasks, encourage users when they do tasks correctly or simply engage in conversation with the user - akin to a human teaching or assisting a user. Our proposed YET to Intervene (YETI) multimodal agent focuses on the research question of identifying circumstances that may require the agent to intervene proactively. This allows the agent to understand when it can intervene in a conversation with human users that can help the user correct mistakes on tasks, like cooking, using AR. Our YETI Agent learns scene understanding signals based on interpretable notions of Structural Similarity (SSIM) on consecutive video frames. We also define the alignment signal which the AI Agent can learn to identify if the video frames corresponding to the user's actions on the task are consistent with expected actions. These signals are used by our AI Agent to determine when it should proactively intervene. We compare our results on the instances of proactive intervention in the HoloAssist multimodal benchmark for an expert agent guiding a user to complete procedural tasks.

Figures (6)

• Figure 1: Overview of the YETI framework detecting the frames of proactive interaction or intervention by a Multimodal AI Agent. Our YETI Agent system generates lightweight features on-the-fly, enabling rapid decision-making for timely user assistance.
• Figure 2: Distribution of Alignment Signal
• Figure 3: Plot of Alignment Signal measuring changing object count along spatio-temporally changing image frames in a video for a procedural task on how to change a mechanical belt. The Expert Agent autonomously intervenes in the 37th second at Frame 37 based on the alignment signal with Frame 36
• Figure 4: Plot of SSIM filtering proactive interventions by expert agents in an image frame (time instance) of a video capturing a procedural task on how to assemble a RAM computer. Autonomous intervention happens at the 98th second in Frame 98
• Figure 5: Intervention Detection for Coffee Making Task.