Grounding Task Assistance with Multimodal Cues from a Single Demonstration

TL;DR

This work addresses the grounding gap in vision-language task understanding by introducing MICA, a multimodal framework that leverages a single demonstration's eye gaze and speech to temporally segment the demonstration, extract keyframes, and generate contextual captions for retrieval-augmented question answering. By combining implicit (gaze) and explicit (speech) cues, MICA significantly outperforms frame-only baselines, with gaze approaching speech performance and their combination yielding the strongest results across tasks like organizing, shopping, and morning routines. The study demonstrates that task type modulates cue effectiveness and highlights the practical value of adaptive multimodal grounding for personalized, real-world AI task assistance. These findings underscore the importance of using rich multimodal context—beyond frames alone—to enhance VLM-based guidance in ecologically valid, single-demonstration scenarios.

Abstract

A person's demonstration often serves as a key reference for others learning the same task. However, RGB video, the dominant medium for representing these demonstrations, often fails to capture fine-grained contextual cues such as intent, safety-critical environmental factors, and subtle preferences embedded in human behavior. This sensory gap fundamentally limits the ability of Vision Language Models (VLMs) to reason about why actions occur and how they should adapt to individual users. To address this, we introduce MICA (Multimodal Interactive Contextualized Assistance), a framework that improves conversational agents for task assistance by integrating eye gaze and speech cues. MICA segments demonstrations into meaningful sub-tasks and extracts keyframes and captions that capture fine-grained intent and user-specific cues, enabling richer contextual grounding for visual question answering. Evaluations on questions derived from real-time chat-assisted task replication show that multimodal cues significantly improve response quality over frame-based retrieval. Notably, gaze cues alone achieves 93% of speech performance, and their combination yields the highest accuracy. Task type determines the effectiveness of implicit (gaze) vs. explicit (speech) cues, underscoring the need for adaptable multimodal models. These results highlight the limitations of frame-based context and demonstrate the value of multimodal signals for real-world AI task assistance.

Figures (16)

• Figure 1: Illustration of multimodal cues in shopping. A user selects a lactose-free product using eye gaze and speech. Gaze highlights the product before speech clarifies intent: "I usually get this to avoid lactose."
• Figure 2: Overview of MICA (Multimodal Interactive Contextualized Assistance). (A) A user demonstrates an activity (e.g., boiling eggs) with multimodal inputs like RGB, speech, and gaze, with key frames highlighted in green. (B) A new user asks context-specific questions (purple: objects, red: actions). (C) MICA extracts contextual cues (e.g., egg count in purple, actions in red) using multimodal signals. (D) MICA provides real-time, personalized instructions, aligning object (purple) and action (red) references with the query.
• Figure 3: MICA uses multimodal cues to extract context from demonstrations. The framework leverages gaze-based cues to segment the demonstration temporally (A) For gaze-driven temporal segmentation, we monitor changes in the user’s gaze to detect and track gaze-fixated objects. Once the set of fixated objects changes substantially, a new temporal segment is started. (B) For each temporal segment, key frames and captions are generated via a Vision-Language Model by conditioning on gaze and/or speech annotations. The resulting information is saved to a database for retrieval-augmented assistance.
• Figure 4: In the demonstration phase (left panels), a user performs a task while wearing a HoloLens 2, recording their eye gaze and speech. During the live evaluation phase (right panels), the system uses the demonstration context to assist a new user in replicating the task by answering real-time questions.
• Figure 5: Performance of base models (GPT4o, GPT4o-mini, VILA, LLaVA) across different context extraction methods (Zero Shot, CLIP Clustering, Frames as Context, Eye Gaze, Speech, Eye Gaze + Speech) on the annotated evaluation set. The plot shows overall model performance across questions. We report mean $\pm$ standard error across evaluation questions (n=415). We include a table version of this plot in Appendix Table \ref{['tab:condition_comparison']}.