SIGN: A Statistically-Informed Gaze Network for Gaze Time Prediction

TL;DR

This work tackles predicting gaze time when only aggregate gaze data are available, proposing SIGN, a Statistically-Informed Gaze Network that fuses a foundational scan-path statistical model with a CNN/Transformer-based neural implementation. The approach decomposes gaze into a gist-driven initial term and a local region contribution, yielding both an overall gaze time predictor and an inferred region-wise gaze pattern $p_{S_j}$. Key contributions include (i) a formal probabilistic formulation linking $\mathbb{E}(G)$ to $(\mu_0, \mu, w_j)$ and (ii) a practical architecture that estimates these components and outputs plausible fixation maps. Empirical results on AdGaze3500 and COCO-Search18 show SIGN achieves better gaze-time prediction than strong baselines and generates gaze maps that align with empirical fixation patterns, demonstrating potential for applications in advertising and human-computer interaction under aggregate data constraints. The work lays groundwork for privacy-preserving gaze modeling with impact potential in industry and research by enabling gaze-time predictions and interpretable fixation maps from aggregate data.

Abstract

We propose a first version of SIGN, a Statistically-Informed Gaze Network, to predict aggregate gaze times on images. We develop a foundational statistical model for which we derive a deep learning implementation involving CNNs and Visual Transformers, which enables the prediction of overall gaze times. The model enables us to derive from the aggregate gaze times the underlying gaze pattern as a probability map over all regions in the image, where each region's probability represents the likelihood of being gazed at across all possible scan-paths. We test SIGN's performance on AdGaze3500, a dataset of images of ads with aggregate gaze times, and on COCO-Search18, a dataset with individual-level fixation patterns collected during search. We demonstrate that SIGN (1) improves gaze duration prediction significantly over state-of-the-art deep learning benchmarks on both datasets, and (2) can deliver plausible gaze patterns that correspond to empirical fixation patterns in COCO-Search18. These results suggest that the first version of SIGN holds promise for gaze-time predictions and deserves further development.

Figures (5)

• Figure 1: Sample predicted weight maps for AdGaze3500. For each sample, the left half presents the original image, and the right half presents the image overlaid with the generated SIGN weights. Brighter regions are predicted to be more conspicuous.
• Figure 2: Sample predicted weight map for COCO-Search18 data. The original image is on the left, the actual fixation locations in the middle (blurred by a Gaussian filter with a standard deviation of 35 pixels bylinskii2018different and resized), and the predicted inferred map is on the right.
• Figure 3: Appendix 1: Diagram of SIGN Architecture. The architecture consists of two separate parts, respectively corresponding to image and context 'gist' $\mu_0$ and total gaze on image regions defined in equation \ref{['eqn:gaze_for_average_equivalent']}. The "gist" is represented by the global visual features extracted by a pre-trained and fine-tunable ResNet50 block, which are then passed to $\mu_0$, modeled by an MLP, to calculate the gaze induced by the "gist". In parallel, the original image is patchified into regions, which are subsequently compressed to local visual features $S_1,S_2,\dots,S_n$ generated by a trainable CNN. Each local feature $S_i$ is used to determine local gaze $\mu_i$ through an MLP; local features are also used to calculate local weights $w_i$, calculated from another MLP on top of a transformer encoder that enhances interactions among local features. The total local gaze pattern is calculated as the dot product between the local gaze and their local weights; the final predicted gaze is the sum of the "gist" gaze and the total local gaze pattern.
• Figure 4: Appendix 2a: Additional Predicted Weight Maps for AdGaze3500.
• Figure 5: Appendix 2b: Additional Predicted Weight Maps for COCO-Search 18.