Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents

TL;DR

This work introduces WAGIBench, the first benchmark for egocentric multimodal goal inference in assistive wearable agents. It provides a large, script-driven dataset with four modalities—vision, audio, digital context, and longitudinal history—collected from 348 participants across 3,477 observation–goal pairs. The evaluation framework combines discriminative (MCQ) and generative (LLM judge) tasks, revealing that humans outperform vision-language models (≈93% vs ≈84% MCQ accuracy) and that larger models improve generative performance but still struggle to reach practical usefulness (≈55% goal relevance). A meta-evaluation shows an LLM Judge, especially when conditioned on reference goals or script cues, can approach human judgment (≈76.8% alignment). The study highlights the importance of relevant modalities and points to future work needed to close the gap toward deployable goal-inference for wearable agents, with implications for accessibility and user experience.

Abstract

There has been a surge of interest in assistive wearable agents: agents embodied in wearable form factors (e.g., smart glasses) who take assistive actions toward a user's goal/query (e.g. "Where did I leave my keys?"). In this work, we consider the important complementary problem of inferring that goal from multi-modal contextual observations. Solving this "goal inference" problem holds the promise of eliminating the effort needed to interact with such an agent. This work focuses on creating WAGIBench, a strong benchmark to measure progress in solving this problem using vision-language models (VLMs). Given the limited prior work in this area, we collected a novel dataset comprising 29 hours of multimodal data from 348 participants across 3,477 recordings, featuring ground-truth goals alongside accompanying visual, audio, digital, and longitudinal contextual observations. We validate that human performance exceeds model performance, achieving 93% multiple-choice accuracy compared with 84% for the best-performing VLM. Generative benchmark results that evaluate several families of modern vision-language models show that larger models perform significantly better on the task, yet remain far from practical usefulness, as they produce relevant goals only 55% of the time. Through a modality ablation, we show that models benefit from extra information in relevant modalities with minimal performance degradation from irrelevant modalities.

Figures (18)

• Figure 1: Three multi-modal samples from the benchmark. In the top row, the video and digital contexts are relevant to the prediction problem, and audio/longitudinal are noise (we call this the $S_\text{VD}$ subset). In the middle row, video and audio are relevant ($S_\text{VA}$). In the bottom row, the video, audio, and longitudinal contexts are relevant ($S_\text{VAL}$).
• Figure 2: (left) Venn diagram showing the spread of recordings where different combinations of modalities are relevant to the goal-inference task (all modalities are always available). Subsets used in our experiments are tagged with the name we use to refer to them ($S_\text{V}$, $S_\text{VA}$, $S_\text{VD}$, $S_\text{VL}$). (right) Histogram of recording locations as estimated by an automated VLM classification.
• Figure 3: Example LLM Judge responses for goal inference predictions, along with reference goals and the judge's reasoning trace. Best viewed when zoomed in.
• Figure 4: (left) Human study subset MCQ accuracy results for humans and the Qwen models, ordered by the mean accuracy of each model across distractor similarities. Error bars indicate bootstrapped 95% confidence intervals of the mean. (right) Mean performance on full dataset with all modality inputs for MCQ and Generative tasks.
• Figure 5: (left) LLM-as-Judge for Generative Evaluation. (right) Alignment between Human raters and Judges with different inductive biases.