Eye Gaze as a Signal for Conveying User Attention in Contextual AI Systems

TL;DR

This work investigates wearable eye tracking as an implicit signal of user attention to improve contextual AI collaboration. It first characterizes the signal quality requirements by analyzing object visual size and ET error using the ADT dataset, deriving practical accuracy thresholds for gaze placement on nearby objects. It then tests how eye-tracking context, in the form of gaze histories (scanpaths), can augment vision-language model queries, showing that prior fixations significantly boost object identification (up to $24.8\%$) and interaction anticipation (up to $49.5\%$) beyond image-only baselines. The findings demonstrate that eye gaze can convey meaningful task-relevant context to AI agents, particularly in near-field scenarios, suggesting a path toward more seamless and private human-AI collaboration—albeit with privacy and edge-case considerations for real-world deployment. These results motivate future work on longer temporal contexts, real-world prototypes, and privacy-preserving designs for gaze-enabled contextual AI systems.

Abstract

Advanced multimodal AI agents can now collaborate with users to solve challenges in the world. Yet, these emerging contextual AI systems rely on explicit communication channels between the user and system. We hypothesize that implicit communication of the user's interests and intent would reduce friction and improve user experience when collaborating with AI agents. In this work, we explore the potential of wearable eye tracking to convey signals about user attention. We measure the eye tracking signal quality requirements to effectively map gaze traces to physical objects, then conduct experiments that provide visual scanpath history as additional context when querying vision language models. Our results show that eye tracking provides high value as a user attention signal and can convey important context about the user's current task and interests, improving understanding of contextual AI agents.

Figures (4)

• Figure 1: Illustration of eye tracking spatial error and object visual size measurements. As an average case measurement, object segmentation area can be mapped to a circular region, with the radius reflecting the eye tracking accuracy requirement (thin bars). Alternatively, $1 / 2$ minor axis span $L_{min}$ (thick bars) is a stricter bound for measuring non-uniform objects.
• Figure 2: Eye tracking accuracy requirements to place gaze accurately within in the ADT dataset, where users performed household actions in an indoor environment pan_adt_2023 Near-field and mid-field measurements consider all objects present in the user's field of view. Interacted objects are being actively manipulated by the user, and fixated objects consider those being directly gazed at.
• Figure 3: Experiments where prior gaze fixation contents are supplied to a VLM along with egocentric images. When many fixations are given as context, the model can synthesize image + gaze information to outperform a greedy baseline that only considers contents from the prompt. The error surfaces in light blue represent 95% confidence intervals.
• Figure 4: An example query to the VLM for E1. The additional context from fixation history is highlighted in red; We vary the amount of context given to the model to measure how this context influences model understanding.