Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding

TL;DR

This work tackles egocentric understanding and short-horizon future action prediction with Vision-Language Models by leveraging human gaze as a training-time regularizer. A modular gaze-regularized attention mechanism aligns model focus with human visual attention, implemented as a KL-divergence penalty between patch-wise model attention and aggregated gaze heatmaps, while allowing RGB-only inference. Across multiple VLM backbones and on Ego4D-derived data, the method yields consistent gains in both current activity understanding and future action prediction, reduces hallucinations, and improves attention alignment with human gaze. The approach demonstrates strong generalization to out-of-distribution egocentric data and maintains practical inference efficiency, paving the way for gaze-grounded, robust assistive and collaborative robotics applications.

Abstract

Eye gaze offers valuable cues about attention, short-term intent, and future actions, making it a powerful signal for modeling egocentric behavior. In this work, we propose a gaze-regularized framework that enhances VLMs for two key egocentric understanding tasks: fine-grained future event prediction and current activity understanding. Unlike prior approaches that rely solely on visual inputs or use gaze as an auxiliary input signal , our method uses gaze only during training. We introduce a gaze-regularized attention mechanism that aligns model focus with human visual gaze. This design is flexible and modular, allowing it to generalize across multiple VLM architectures that utilize attention. Experimental results show that our approach improves semantic prediction scores by up to 11 for future event prediction and around 7 for current activity understanding, compared to the corresponding baseline models trained without gaze regularization. These results highlight the value of gaze-guided training in improving the accuracy and robustness of egocentric VLMs. Overall, this work establishes a foundation for using human gaze to enhance the predictive capabilities of VLMs in real-world scenarios like assistive robots and human-machine collaboration. Code and additional information is available at: https://github.com/anupampani/Gaze-VLM

Figures (10)

• Figure 1: Overview of the Proposed Scheme. During inference ( top), egocentric frames are processed by a vision encoder to extract visual tokens, which flow through a transformer encoder to produce aggregated features that are decoded into descriptive text. During training ( bottom), we add gaze regularization: human gaze heatmaps are converted to patch-wise distributions matching the transformer's attention granularity. The model's attention distribution is then guided toward human visual focus patterns through KL-divergence minimization, enhancing feature aggregation without requiring gaze data during inference.
• Figure 2: Temporal Aggregation of Gaze.Top: Input egocentric frames. Bottom: Gaze points transformed into heatmaps, filtered using bidirectional optical flow ($f_{\tau \rightarrow t}$, $f_{t \rightarrow \tau}$) to remove occluded points, then warped and aggregated into a temporally coherent supervision signal.
• Figure 3: Qualitative Results of Future Activity Prediction. Examples comparing predictions from the base model and our gaze-regularized model against ground-truth annotations and actual future frames. The gaze-regularized model correctly predicts specific objects and actions (e.g., "picking up chocolate") while the base model makes less accurate predictions (e.g., incorrectly predicting "a drink"). Key words in the predictions are highlighted to emphasize differences in prediction specificity and accuracy.
• Figure 4: Comparison of Attention Maps. Attention distributions from base model (without gaze regularization) versus our gaze-regularized model. Our approach produces more semantically meaningful attention: Top: Our model focuses on specific objects (cards, book, phone) while the base model attends broadly to the table. Middle: Our model attends to both the person and plastic packet, compared to the base model's person-only focus. Bottom: Our model correctly attends to chocolate and drink on the counter, while the base model attends to less relevant areas. This demonstrates how gaze regularization guides the model to focus on objects crucial for understanding human activities.
• Figure 5: GPT-4V Annotation WorkflowLeft. We illustrate our annotation pipeline, where sequences of egocentric images are provided to GPT-4V along with an iteratively refined prompt. Human feedback is used to improve the prompt over multiple rounds, ensuring greater accuracy and contextual coherence in the generated captions. The use of the entire sequence helps maintain temporal consistency, allowing GPT-4V to better capture object interactions and ongoing actions. Right. We show a sample image from the dataset, the final refined prompt, and the resulting annotation. This process results in finer-grained text annotations which we utilise in our experiments.