Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GazeProphet: Software-Only Gaze Prediction for VR Foveated Rendering

Farhaan Ebadulla, Chiraag Mudlapur, Gaurav BV

TL;DR

GazeProphet presents a software-only solution for predicting gaze in VR to enable foveated rendering without eye-tracking hardware. It fuses spatial features from a Spherical Vision Transformer with temporal gaze dynamics via an LSTM temporal encoder, using a multi-modal fusion network to output gaze coordinates and a confidence score. The approach achieves a median angular error of $3.83^\circ$, outperforming saliency baselines by $24\%$, and demonstrates robust performance across regions and scenes with strong statistical significance. This work lowers barriers to widespread foveated rendering, enabling VR optimization on existing hardware and broadening accessibility across platforms.

Abstract

Foveated rendering significantly reduces computational demands in virtual reality applications by concentrating rendering quality where users focus their gaze. Current approaches require expensive hardware-based eye tracking systems, limiting widespread adoption due to cost, calibration complexity, and hardware compatibility constraints. This paper presents GazeProphet, a software-only approach for predicting gaze locations in VR environments without requiring dedicated eye tracking hardware. The approach combines a Spherical Vision Transformer for processing 360-degree VR scenes with an LSTM-based temporal encoder that captures gaze sequence patterns. A multi-modal fusion network integrates spatial scene features with temporal gaze dynamics to predict future gaze locations with associated confidence estimates. Experimental evaluation on a comprehensive VR dataset demonstrates that GazeProphet achieves a median angular error of 3.83 degrees, outperforming traditional saliency-based baselines by 24% while providing reliable confidence calibration. The approach maintains consistent performance across different spatial regions and scene types, enabling practical deployment in VR systems without additional hardware requirements. Statistical analysis confirms the significance of improvements across all evaluation metrics. These results show that software-only gaze prediction can work for VR foveated rendering, making this performance boost more accessible to different VR platforms and apps.

GazeProphet: Software-Only Gaze Prediction for VR Foveated Rendering

TL;DR

GazeProphet presents a software-only solution for predicting gaze in VR to enable foveated rendering without eye-tracking hardware. It fuses spatial features from a Spherical Vision Transformer with temporal gaze dynamics via an LSTM temporal encoder, using a multi-modal fusion network to output gaze coordinates and a confidence score. The approach achieves a median angular error of , outperforming saliency baselines by , and demonstrates robust performance across regions and scenes with strong statistical significance. This work lowers barriers to widespread foveated rendering, enabling VR optimization on existing hardware and broadening accessibility across platforms.

Abstract

Foveated rendering significantly reduces computational demands in virtual reality applications by concentrating rendering quality where users focus their gaze. Current approaches require expensive hardware-based eye tracking systems, limiting widespread adoption due to cost, calibration complexity, and hardware compatibility constraints. This paper presents GazeProphet, a software-only approach for predicting gaze locations in VR environments without requiring dedicated eye tracking hardware. The approach combines a Spherical Vision Transformer for processing 360-degree VR scenes with an LSTM-based temporal encoder that captures gaze sequence patterns. A multi-modal fusion network integrates spatial scene features with temporal gaze dynamics to predict future gaze locations with associated confidence estimates. Experimental evaluation on a comprehensive VR dataset demonstrates that GazeProphet achieves a median angular error of 3.83 degrees, outperforming traditional saliency-based baselines by 24% while providing reliable confidence calibration. The approach maintains consistent performance across different spatial regions and scene types, enabling practical deployment in VR systems without additional hardware requirements. Statistical analysis confirms the significance of improvements across all evaluation metrics. These results show that software-only gaze prediction can work for VR foveated rendering, making this performance boost more accessible to different VR platforms and apps.

Paper Structure

This paper contains 30 sections, 10 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Eye Tracking in VR Systems
  4. Foveated Rendering Approaches
  5. Gaze Prediction and Saliency Models
  6. Vision Transformers and Attention Mechanisms
  7. Temporal Sequence Modeling
  8. Approach
  9. Architecture Overview
  10. Spherical Vision Transformer
  11. Patch Embedding Process
  12. Positional Encoding Implementation
  13. Multi-Head Attention Processing
  14. LSTM Temporal Encoder
  15. Input Sequence Preparation
  16. ...and 15 more sections

Figures (3)

  • Figure 1: GazeProphet Architecture: Multi-modal approach combining Spherical Vision Transformer for 360° scene processing, LSTM Temporal Encoder for gaze sequence modeling, and Multi-Modal Fusion Network for final prediction generation.
  • Figure 2: Spatial error distribution heatmaps comparing GazeProphet against baseline approaches. GazeProphet maintains consistent performance across all regions while baselines show center bias.
  • Figure 3: Sample prediction comparisons across three VR scenes showing ground truth gaze locations versus model predictions with confidence scores. GazeProphet achieves superior accuracy with high confidence scores compared to baseline approaches.