Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Spatio-Temporal Feature Representations for Video-Based Gaze Estimation

Alexandre Personnic, Mihai Bâce

TL;DR

ST-Gaze addresses video-based gaze estimation by preserving intra-frame spatial context before temporal fusion, leveraging a dual-stream CNN backbone, channel attention, and a spatio-temporal recurrence. The method achieves state-of-the-art single-view 3D gaze accuracy on the EVE dataset (2.58°) and provides a strong backbone for downstream person-specific adaptation (1.87° offline with SCPT). An extensive ablation study validates the importance of the Self-Attention Module and the spatio-temporal recurrence, highlighting the superiority of maintaining spatial information over premature pooling. The approach demonstrates robust, camera-only gaze estimation in real-world settings and offers a principled path toward more general-purpose video gaze systems and downstream personalization.

Abstract

Video-based gaze estimation methods aim to capture the inherently temporal dynamics of human eye gaze from multiple image frames. However, since models must capture both spatial and temporal relationships, performance is limited by the feature representations within a frame but also between multiple frames. We propose the Spatio-Temporal Gaze Network (ST-Gaze), a model that combines a CNN backbone with dedicated channel attention and self-attention modules to fuse eye and face features optimally. The fused features are then treated as a spatial sequence, allowing for the capture of an intra-frame context, which is then propagated through time to model inter-frame dynamics. We evaluated our method on the EVE dataset and show that ST-Gaze achieves state-of-the-art performance both with and without person-specific adaptation. Additionally, our ablation study provides further insights into the model performance, showing that preserving and modelling intra-frame spatial context with our spatio-temporal recurrence is fundamentally superior to premature spatial pooling. As such, our results pave the way towards more robust video-based gaze estimation using commonly available cameras.

Learning Spatio-Temporal Feature Representations for Video-Based Gaze Estimation

TL;DR

ST-Gaze addresses video-based gaze estimation by preserving intra-frame spatial context before temporal fusion, leveraging a dual-stream CNN backbone, channel attention, and a spatio-temporal recurrence. The method achieves state-of-the-art single-view 3D gaze accuracy on the EVE dataset (2.58°) and provides a strong backbone for downstream person-specific adaptation (1.87° offline with SCPT). An extensive ablation study validates the importance of the Self-Attention Module and the spatio-temporal recurrence, highlighting the superiority of maintaining spatial information over premature pooling. The approach demonstrates robust, camera-only gaze estimation in real-world settings and offers a principled path toward more general-purpose video gaze systems and downstream personalization.

Abstract

Video-based gaze estimation methods aim to capture the inherently temporal dynamics of human eye gaze from multiple image frames. However, since models must capture both spatial and temporal relationships, performance is limited by the feature representations within a frame but also between multiple frames. We propose the Spatio-Temporal Gaze Network (ST-Gaze), a model that combines a CNN backbone with dedicated channel attention and self-attention modules to fuse eye and face features optimally. The fused features are then treated as a spatial sequence, allowing for the capture of an intra-frame context, which is then propagated through time to model inter-frame dynamics. We evaluated our method on the EVE dataset and show that ST-Gaze achieves state-of-the-art performance both with and without person-specific adaptation. Additionally, our ablation study provides further insights into the model performance, showing that preserving and modelling intra-frame spatial context with our spatio-temporal recurrence is fundamentally superior to premature spatial pooling. As such, our results pave the way towards more robust video-based gaze estimation using commonly available cameras.

Paper Structure

This paper contains 32 sections, 3 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Appearance-based Gaze Estimation.
  4. Temporal Models in Gaze Estimation.
  5. Attention and Transformers for Gaze Estimation.
  6. Spatio-Temporal Gaze Network (ST-Gaze) Architecture
  7. Multi-Stream Feature Extraction
  8. Attention-Based Feature Fusion
  9. Channel-wise Fusion.
  10. Spatial Feature Learning.
  11. Spatio-Temporal Recurrence
  12. Gaze Regression and Loss Function
  13. Loss Function.
  14. Experiments
  15. Setup
  16. ...and 17 more sections

Figures (4)

  • Figure 1: The architecture of our proposed ST-Gaze. An input frame is processed by parallel eye and face encoders. The features are fused via ECA and a Self-Attention Module (SAM). A GRU first processes the feature maps spatially (intra-frame) before propagating its hidden state temporally (inter-frame) to the next frame. A final multi-layer perceptron (MLP) regresses the gaze vector. Input from EVE parkEndtoendVideobasedEyeTracking2020.
  • Figure 2: Analysis of ST-Gaze's robustness on the EVE test set. (a) Performance is stable across stimuli and camera views, with the centre webcam (webcam_c) showing the lowest variance. (b) Performance varies significantly across participants, highlighting the challenge of person-specific factors.
  • Figure 3: Qualitative examples from the EVE test set parkEndtoendVideobasedEyeTracking2020. The four leftmost frames show a challenging case (Participant test03) while the rightmost frame shows a high-performance case (Participant test04). Images reproduced with permission from authors.
  • Figure 4: Detailed performance analysis of ST-Gaze on the EVE validation set. These figures provide a granular breakdown of the aggregate results presented in the main paper's ablation study.