ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimation

TL;DR

This work tackles online egocentric gaze estimation by reframing it as causal autoregressive sequence generation. ARGaze integrates a Scene Encoder, Autoregressive Heatmap Tokenizer, Tracking-aware Template Module, and a causal Transformer Decoder to predict gaze heatmaps $\mathbf{H}_t$ from current visual input and a bounded history $\mathbf{H}_{t-k:t-1}$, enabling strictly online inference with a fixed memory footprint. The model achieves state-of-the-art performance on EGTEA Gaze+, Ego4D, and EgoExo4D, improving robustness to hand-induced biases and generalizing to out-of-distribution scenarios, while significantly boosting inference speed (e.g., up to $1.82\times$) and reducing memory usage. Key contributions include the explicit autoregressive formulation with a bounded gaze-history window, a heatmap tokenizer for temporal tokens, a tracking-aware template mechanism for localized priors, and a training regime using KL loss with scheduled sampling, culminating in real-world streaming viability for AR and assistive technologies. These advances offer practical impact for real-time gaze-aware systems, while also raising privacy considerations and the need for responsible deployment in user-facing applications $ $.

Abstract

Online egocentric gaze estimation predicts where a camera wearer is looking from first-person video using only past and current frames, a task essential for augmented reality and assistive technologies. Unlike third-person gaze estimation, this setting lacks explicit head or eye signals, requiring models to infer current visual attention from sparse, indirect cues such as hand-object interactions and salient scene content. We observe that gaze exhibits strong temporal continuity during goal-directed activities: knowing where a person looked recently provides a powerful prior for predicting where they look next. Inspired by vision-conditioned autoregressive decoding in vision-language models, we propose ARGaze, which reformulates gaze estimation as sequential prediction: at each timestep, a transformer decoder predicts current gaze by conditioning on (i) current visual features and (ii) a fixed-length Gaze Context Window of recent gaze target estimates. This design enforces causality and enables bounded-resource streaming inference. We achieve state-of-the-art performance across multiple egocentric benchmarks under online evaluation, with extensive ablations validating that autoregressive modeling with bounded gaze history is critical for robust prediction. We will release our source code and pre-trained models.

Figures (10)

• Figure 1: Comparison between offline clip-based and online autoregressive gaze estimation. Traditional methods (top) rely on 3D convolutions or bidirectional attention, requiring future context that is impractical for real-time deployment. In contrast, ARGaze reformulates gaze estimation as a causal sequence generation task. By utilizing a visual scene encoder and conditioning on History Heatmap Tokens for temporal stability, ARGaze achieves strictly online inference with constant memory overhead, ensuring both responsiveness to new visual observations and fixation stability.
• Figure 2: Overview of the ARGaze framework for online egocentric gaze estimation. Our model treats gaze prediction as an autoregressive sequence generation task. At each timestep $t$, the Scene Encoder extracts features from the current frame $\mathbf{I}_t$, while the Tracking-aware Template Module provides localized visual priors from previous fixations. The Autoregressive Heatmap Tokenizer converts historical heatmaps $\mathbf{H}_{t-k:t-1}$ into temporal tokens. These inputs are integrated by a Transformer Decoder and mapped back to the spatial domain via a Reconstruction Head to produce the final gaze target heatmap $\mathbf{H}_t$.
• Figure 3: Autoregressive Decoder. The decoder takes $\mathbf{S}_{tgt}=[\mathbf{H}_{t-k:t-1}, \mathbf{q}_t]$ (shown with $k=3$) and applies $N$ masked self-attention and cross-attention layers over visual features (current frame + tracking-aware template embeddings) to predict heatmap tokens with strict causality and constant memory.
• Figure 4: ARGaze and GLC Regarding Hand-Bias. We can see that the GLC suffers from significant hand-bias as the predicted gaze erroneously tracks the moving hand. In contrast, ARGaze maintains object-centric stability. The red shaded area highlights a period of intensive hand-object interaction where our structured dynamical model ensures a consistent fixation on the task-relevant object. Note that Exo (third-person) frames are not available to the model for egocentric gaze estimation and are shown only for visualization purposes.
• Figure 5: Comparison of gaze heatmaps across methods during target shifts and new fixations. ARGaze fuses visual cues with the historical continuity of the gaze trajectory, enabling rapid adaptation during target shift and stable fixation on task-relevant objects.