HOIGaze: Gaze Estimation During Hand-Object Interactions in Extended Reality Exploiting Eye-Hand-Head Coordination

TL;DR

HOIGaze introduces a hierarchical gaze estimation framework tailored for hand-object interactions in XR, exploiting tight eye–hand–head coordination to denoise training samples. The method combines an attended-hand recogniser (CNN + ST-GCN) with a cross-modal gaze estimator (CNN + ST-GCN + cross-modal Transformers) and introduces an eye-head coordination loss to emphasize coordinative samples. Empirical results on HOT3D and ADT show substantial improvements in mean angular error over state-of-the-art baselines, with up to 15.6% average gains in HOI scenarios and meaningful gains in mixed settings, plus improved eye-based activity recognition downstream performance. The work demonstrates the rich informational content of eye-hand-head coordination for gaze estimation and suggests practical applicability in XR systems and related tasks.

Abstract

We present HOIGaze - a novel learning-based approach for gaze estimation during hand-object interactions (HOI) in extended reality (XR). HOIGaze addresses the challenging HOI setting by building on one key insight: The eye, hand, and head movements are closely coordinated during HOIs and this coordination can be exploited to identify samples that are most useful for gaze estimator training - as such, effectively denoising the training data. This denoising approach is in stark contrast to previous gaze estimation methods that treated all training samples as equal. Specifically, we propose: 1) a novel hierarchical framework that first recognises the hand currently visually attended to and then estimates gaze direction based on the attended hand; 2) a new gaze estimator that uses cross-modal Transformers to fuse head and hand-object features extracted using a convolutional neural network and a spatio-temporal graph convolutional network; and 3) a novel eye-head coordination loss that upgrades training samples belonging to the coordinated eye-head movements. We evaluate HOIGaze on the HOT3D and Aria digital twin (ADT) datasets and show that it significantly outperforms state-of-the-art methods, achieving an average improvement of 15.6% on HOT3D and 6.0% on ADT in mean angular error. To demonstrate the potential of our method, we further report significant performance improvements for the sample downstream task of eye-based activity recognition on ADT. Taken together, our results underline the significant information content available in eye-hand-head coordination and, as such, open up an exciting new direction for learning-based gaze estimation.

Figures (3)

• Figure 1: HOIGaze combines an attended hand recogniser and an eye gaze estimator. The attended hand recogniser uses a 1D CNN and two ST-GCNs to extract features from head orientations, left and right hand gestures, respectively, to recognise the attended hand. The gaze estimator uses an ST-GCN to extract features from the attended hand and scene objects, and then fuses the head and hand-object features using cross-modal Transformers to estimate eye gaze.
• Figure 2: The cumulative distribution functions of different methods' estimation errors on HOT3D (Cross-User), HOT3D (Cross-Scene), and ADT. The higher the CDF curve, the better the performance. Our method achieves better performance than other methods in terms of estimation error distributions.
• Figure 3: Visualisation of the gaze estimation results from our method and the state-of-the-art method Pose2Gazehu24pose2gaze on the HOT3D dataset. The green arrow represents the ground truth eye gaze, the red arrow denotes our method, the blue arrow refers to Pose2Gaze. Our method exhibits higher estimation accuracy than the state-of-the-art method at different scenarios and different activities.