HAGI++: Head-Assisted Gaze Imputation and Generation

TL;DR

HAGI++ tackles the pervasive problem of missing gaze data in mobile eye tracking by introducing a transformer-based diffusion model conditioned on time-aligned head motion, and optionally wrist/hand signals, to impute or generate gaze trajectories. By explicitly modeling eye-head (and eye-head-hand) coordination with FiLM-based fusion, it achieves consistent improvements over interpolation and prior diffusion baselines, reducing mean angular error and producing more realistic gaze velocity distributions. The approach demonstrates strong generalisation across Nymeria, Ego-Exo4D, and HOT3D datasets and extends to gaze generation under 100% data loss, outperforming full-body-motion baselines using only commodity wearables. The work offers practical pathways for higher-quality gaze data in real-world and XR applications, enabling better gaze analytics, interactive systems, and synthetic eye motion for graphics.

Abstract

Mobile eye tracking plays a vital role in capturing human visual attention across both real-world and extended reality (XR) environments, making it an essential tool for applications ranging from behavioural research to human-computer interaction. However, missing values due to blinks, pupil detection errors, or illumination changes pose significant challenges for further gaze data analysis. To address this challenge, we introduce HAGI++ - a multi-modal diffusion-based approach for gaze data imputation that, for the first time, uses the integrated head orientation sensors to exploit the inherent correlation between head and eye movements. HAGI++ employs a transformer-based diffusion model to learn cross-modal dependencies between eye and head representations and can be readily extended to incorporate additional body movements. Extensive evaluations on the large-scale Nymeria, Ego-Exo4D, and HOT3D datasets demonstrate that HAGI++ consistently outperforms conventional interpolation methods and deep learning-based time-series imputation baselines in gaze imputation. Furthermore, statistical analyses confirm that HAGI++ produces gaze velocity distributions that closely match actual human gaze behaviour, ensuring more realistic gaze imputations. Moreover, by incorporating wrist motion captured from commercial wearable devices, HAGI++ surpasses prior methods that rely on full-body motion capture in the extreme case of 100% missing gaze data (pure gaze generation). Our method paves the way for more complete and accurate eye gaze recordings in real-world settings and has significant potential for enhancing gaze-based analysis and interaction across various application domains.

Figures (4)

• Figure 1: Overview of the training pipeline and model architecture of HAGI++. During training, a complete gaze sequence is divided into an observed segment and a target segment. The objective of HAGI++ is to reconstruct the target gaze sequence from the observed gaze, time-aligned head movements, and optionally, other body movements (e.g., wrist motions) captured by commodity wearable devices. All input modalities are projected into token representations via MLPs. The core of HAGI++ consists of a stack of $N$ transformer blocks. Each block contains a self-attention layer that captures correlations between the concatenated observed and noisy gaze tokens, a cross-attention layer that models eye--head or eye--hand--head coordination, and a FiLM layer that further integrates movement features for gaze prediction. HAGI++ can be readily adapted for gaze generation by removing the observed gaze input. This design enables HAGI++ to effectively capture multimodal eye--head or eye--hand--head coordination for accurate and realistic gaze imputation and generation.
• Figure 2: Four examples of gaze imputation results at different missing ratios (10%, 30%, 50%, 90%) using different methods in the cross-dataset evaluation. The bottom row shows the visualisations of ground truth human eye movements.
• Figure 3: Visualisation of the gaze generation results of one-point, two-point, three-point HAGI++ and the state-of-the-art method Pose2Gaze hu24pose2gaze that repuires full-body pose as input on the Nymeria dataset. Our method exhibits higher gaze generation accuracy with input from only commodity wearable devices. More videos can be found in our supplementary materials.
• Figure :