Saccade-Contingent Rendering

TL;DR

This work introduces a non-foveated, saccade-contingent rendering method that reduces render resolution for a brief post-saccadic window to exploit reduced acuity after gaze shifts, without requiring precise eye tracking. The authors characterize post-saccadic acuity over time with psychophysics, derive a temporally varying render-accuracy curve $sf(t)$, and validate the approach through benchtop and in-headset experiments, including a 90 ppd headset with ~30 ms eye-to-photon latency. Results show measurable bitrate savings at high display resolutions and that downsampling can remain perceptually imperceptible under appropriate timing and frequency parameters, enabling meaningful data and power reductions in VR rendering. The method can complement existing perceptual techniques, tolerate limited eye-tracking precision, and extend to future varifocal designs and alternative eye-tracking technologies, offering a practical path to faster, more efficient VR displays without compromising perceived image quality.

Abstract

Battery-constrained power consumption, compute limitations, and high frame rate requirements in head-mounted displays present unique challenges in the drive to present increasingly immersive and comfortable imagery in virtual reality. However, humans are not equally sensitive to all regions of the visual field, and perceptually-optimized rendering techniques are increasingly utilized to address these bottlenecks. Many of these techniques are gaze-contingent and often render reduced detail away from a user's fixation. Such techniques are dependent on spatio-temporally-accurate gaze tracking and can result in obvious visual artifacts when eye tracking is inaccurate. In this work we present a gaze-contingent rendering technique which only requires saccade detection, bypassing the need for highly-accurate eye tracking. In our first experiment, we show that visual acuity is reduced for several hundred milliseconds after a saccade. In our second experiment, we use these results to reduce the rendered image resolution after saccades in a controlled psychophysical setup, and find that observers cannot discriminate between saccade-contingent reduced-resolution rendering and full-resolution rendering. Finally, in our third experiment, we introduce a 90 pixels per degree headset and validate our saccade-contingent rendering method under typical VR viewing conditions.

Figures (6)

• Figure 1: (a) Stimulus presentation for Experiment 1. Participant spatial acuity is determined as a function of post-saccadic duration with a Gabor filter orientation-identification task. (b) Acuity thresholds as a function of time after saccade landing. Colored lines show individual threshold curves, an example data set used to derive each participants' curve is shown in Figure \ref{['fig:teaser']}b. Data for all participants can be found in the supplementary materials. The thick gray line denotes the power fit to the median of all participants' acuity thresholds (R$^2$=0.71).
• Figure 2: Relative bitrate savings shown for a 90 Hz, 32-bit RGB display for 30-100 ppd native resolution displays and 1-8 Hz saccade frequencies. Higher saccade rates enable more-frequent saccade-contingent rendering and greatly increases bit savings as the most significant savings occur in the first 100 ms following saccade landing. Data collection from Figure \ref{['fig:exp1_results']} is limited to 27 cpd and 500 ms, to account for these limitations in our simulation we revert to rendering at native-resolution after 333 ms. Most of the bit savings are achieved in the first 200 ms of our algorithm, so any cutoff >200 ms does not significantly influence this simulation $(\approx\pm20\%)$.
• Figure 3: (a) Thirty different images used in Experiment 2. (b) Likert rating responses to how strong the resolution change was perceived, as a function of when the change occurred after saccade landing. Larger numbers indicate the change was perceived to a greater extent. (c) Likert rating responses analyzed into the probability of detecting changes. (b-c) Horizontal bars on the top indicate the baseline condition in which there was no change in rendered resolution. The color and size of the circles and bars reflect the Likert rating and probability numbers. The gray solid line indicates the group acuity thresholds from Experiment 1. Statistical significance from the baseline condition are denoted in asterisks. ***p<0.001, n.s. not significant.
• Figure 4: Exploded view of our 90 ppd HMD used to run Experiment 3. This pixel density is achieved by significantly reducing the HMD field of view. This trade-off is made to assess the perceptual consequences of downsampling artifacts from saccade-contingent rendering in a scenario limited by human visual acuity rather than display pixel size. We note that analysis from Section \ref{['sec:bandwidth']} identifies potential bitrate savings of 80% at 90 ppd.
• Figure 5: Saccade-contignent rendering downsampling curves evaluated in Experiment 3.