DHECA-SuperGaze: Dual Head-Eye Cross-Attention and Super-Resolution for Unconstrained Gaze Estimation

TL;DR

The paper tackles unconstrained gaze estimation in the wild, addressing challenges from low-resolution inputs and head-eye interactions. It proposes DHECA-SuperGaze, a dual-branch architecture that applies image super-resolution to head images and computes multiscale head features alongside eye crops, enhanced by a dual head-eye cross-attention (DHECA) module. Gaze is represented via sine and cosine components of yaw and pitch, $(s_y,c_y,s_p)$, with yaw estimated as $\theta = w \cdot \theta_S + (1-w) \cdot \theta_C$ where $w = |\cos((\theta_S+\theta_C)/2)|$, and $\theta_S=\sin^{-1}(s_y)$, $\theta_C=\cos^{-1}(c_y)$. Experiments on Gaze360 and GFIE show state-of-the-art within- and cross-dataset performance, and the authors demonstrate that rectifying Gaze360 annotations further improves results.

Abstract

Unconstrained gaze estimation is the process of determining where a subject is directing their visual attention in uncontrolled environments. Gaze estimation systems are important for a myriad of tasks such as driver distraction monitoring, exam proctoring, accessibility features in modern software, etc. However, these systems face challenges in real-world scenarios, partially due to the low resolution of in-the-wild images and partially due to insufficient modeling of head-eye interactions in current state-of-the-art (SOTA) methods. This paper introduces DHECA-SuperGaze, a deep learning-based method that advances gaze prediction through super-resolution (SR) and a dual head-eye cross-attention (DHECA) module. Our dual-branch convolutional backbone processes eye and multiscale SR head images, while the proposed DHECA module enables bidirectional feature refinement between the extracted visual features through cross-attention mechanisms. Furthermore, we identified critical annotation errors in one of the most diverse and widely used gaze estimation datasets, Gaze360, and rectified the mislabeled data. Performance evaluation on Gaze360 and GFIE datasets demonstrates superior within-dataset performance of the proposed method, reducing angular error (AE) by 0.48° (Gaze360) and 2.95° (GFIE) in static configurations, and 0.59° (Gaze360) and 3.00° (GFIE) in temporal settings compared to prior SOTA methods. Cross-dataset testing shows improvements in AE of more than 1.53° (Gaze360) and 3.99° (GFIE) in both static and temporal settings, validating the robust generalization properties of our approach.

Figures (7)

• Figure 1: The main idea behind DHECA-SuperGaze. The method leverages super-resolution (SR) and dual cross-feature attention between head and eye visual features to estimate the gaze direction.
• Figure 2: Distribution of unique face locations across images from the train (left), validation (center), and test (right) subsets of the Gaze360 dataset. Red rectangles encapsulate areas within which the annotations are valid.
• Figure 3: Annotations from the original (top) and rectified (bottom) Gaze360 dataset. Red and yellow rectangles mark the position of the face and eyes, respectively. The original annotations were kept in the two left columns and rectified in the two right columns.
• Figure 4: Scheme of the proposed method, divided into input pre-processing (brown rounded rectangle) and gaze estimation model (orange rounded rectangle). Within the proposed model, the dual cross-attention block and the DHECA module are marked with red and green dashed rounded rectangles, respectively.
• Figure 5: Example of input pre-processing in static and temporal settings. In addition to applying eye detection to input head image/s, the input is passed through an SR model, whose result is eventually rescaled using particular scales.