Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gaze Prediction in Virtual Reality Without Eye Tracking Using Visual and Head Motion Cues

Christos Petrou, Harris Partaourides, Athanasios Balomenos, Yannis Kopsinis, Sotirios Chatzis

TL;DR

This work addresses gaze prediction in VR when direct eye tracking is unavailable due to hardware or privacy constraints by fusing real-time HMD motion signals with visual saliency from a lightweight UniSal encoder. It evaluates two time-series predictors, LSTM and TSMixer, on the EHTask dataset and demonstrates meaningful gaze forecasting improvements over Center-of-HMD baselines, achieving accurate short-term predictions within $q=10$ steps (~$333~\mathrm{ms}$) and extending to $q=30$ steps (~$1000~\mathrm{ms}$). The approach is validated across Linux and Apple ecosystems, with on-device CoreML deployment showing real-time viability, thereby enabling privacy-preserving, attention-aware VR techniques such as foveated rendering. Overall, the framework offers a practical, deployable solution for reducing perceptual lag in VR under eye-tracking constraints.

Abstract

Gaze prediction plays a critical role in Virtual Reality (VR) applications by reducing sensor-induced latency and enabling computationally demanding techniques such as foveated rendering, which rely on anticipating user attention. However, direct eye tracking is often unavailable due to hardware limitations or privacy concerns. To address this, we present a novel gaze prediction framework that combines Head-Mounted Display (HMD) motion signals with visual saliency cues derived from video frames. Our method employs UniSal, a lightweight saliency encoder, to extract visual features, which are then fused with HMD motion data and processed through a time-series prediction module. We evaluate two lightweight architectures, TSMixer and LSTM, for forecasting future gaze directions. Experiments on the EHTask dataset, along with deployment on commercial VR hardware, show that our approach consistently outperforms baselines such as Center-of-HMD and Mean Gaze. These results demonstrate the effectiveness of predictive gaze modeling in reducing perceptual lag and enhancing natural interaction in VR environments where direct eye tracking is constrained.

Gaze Prediction in Virtual Reality Without Eye Tracking Using Visual and Head Motion Cues

TL;DR

This work addresses gaze prediction in VR when direct eye tracking is unavailable due to hardware or privacy constraints by fusing real-time HMD motion signals with visual saliency from a lightweight UniSal encoder. It evaluates two time-series predictors, LSTM and TSMixer, on the EHTask dataset and demonstrates meaningful gaze forecasting improvements over Center-of-HMD baselines, achieving accurate short-term predictions within steps (~) and extending to steps (~). The approach is validated across Linux and Apple ecosystems, with on-device CoreML deployment showing real-time viability, thereby enabling privacy-preserving, attention-aware VR techniques such as foveated rendering. Overall, the framework offers a practical, deployable solution for reducing perceptual lag in VR under eye-tracking constraints.

Abstract

Gaze prediction plays a critical role in Virtual Reality (VR) applications by reducing sensor-induced latency and enabling computationally demanding techniques such as foveated rendering, which rely on anticipating user attention. However, direct eye tracking is often unavailable due to hardware limitations or privacy concerns. To address this, we present a novel gaze prediction framework that combines Head-Mounted Display (HMD) motion signals with visual saliency cues derived from video frames. Our method employs UniSal, a lightweight saliency encoder, to extract visual features, which are then fused with HMD motion data and processed through a time-series prediction module. We evaluate two lightweight architectures, TSMixer and LSTM, for forecasting future gaze directions. Experiments on the EHTask dataset, along with deployment on commercial VR hardware, show that our approach consistently outperforms baselines such as Center-of-HMD and Mean Gaze. These results demonstrate the effectiveness of predictive gaze modeling in reducing perceptual lag and enhancing natural interaction in VR environments where direct eye tracking is constrained.
Paper Structure (10 sections, 17 equations, 4 figures, 2 tables)

This paper contains 10 sections, 17 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Multi-Modal Data Preparation
  5. Time-Series Prediction Model Training
  6. Experiments
  7. Temporal Sequence Modeling
  8. Performance on Apple Ecosystem
  9. Conclusion
  10. Acknowledgements

Figures (4)

  • Figure 1: Example of gaze prediction on the EHTask dataset. Blue: HMD center; green: true gaze; red: predicted gaze using the proposed method.
  • Figure 2: Overview of the multi-modal gaze prediction process
  • Figure 3: Test set evaluation of gaze prediction over a 333 ms horizon. Results for LSTM and TSMixer are compared with the center-of-HMD baseline: azimuth, elevation, combined axes, and relative improvement over the baseline.
  • Figure 4: Test set evaluation of gaze prediction over a 1000 ms horizon. Results for LSTM and TSMixer are compared with the center-of-HMD baseline: azimuth, elevation, combined axes, and relative improvement over the baseline.