Enhancing Foveated Rendering with Weighted Reservoir Sampling

TL;DR

The paper addresses the inefficiency of traditional foveated rendering which discards high-spatial-frequency samples as gaze shifts. It introduces Weighted Reservoir Sampling (WRS) to temporally accumulate perceptually relevant high-quality pixel samples across frames, using a Bernoulli-based temporal bias to discard stale samples. The method maintains a per-pixel reservoir (conceptually of size $n=1$) and combines reservoirs across time to reconstruct higher perceptual quality while enabling higher foveation, running in under $1$ ms at $4$K and integrating with real-time VR/AR pipelines. Empirical evaluations, including text-reading and perceptual validation studies, demonstrate improved image quality and reading performance, with favorable runtime characteristics compared to standard foveated rendering and Temporal Anti-Aliasing (TAA).

Abstract

Spatiotemporal sensitivity to high frequency information declines with increased peripheral eccentricity. Foveated rendering exploits this by decreasing the spatial resolution of rendered images in peripheral vision, reducing the rendering cost by omitting high frequency details. As foveation levels increase, the rendering quality is reduced, and traditional foveated rendering systems tend not to preserve samples that were previously rendered at high spatial resolution in previous frames. Additionally, prior research has shown that saccade landing positions are distributed around a target location rather than landing at a single point, and that even during fixations, eyes perform small microsaccades around a fixation point. This creates an opportunity for sampling from temporally neighbouring frames with differing foveal locations to reduce the required rendered size of the foveal region while achieving a higher perceived image quality. We further observe that the temporal presentation of pixels frame-to-frame can be viewed as a data stream, presenting a random sampling problem. Following this intuition, we propose a Weighted Reservoir Sampling technique to efficiently maintain a reservoir of the perceptually relevant high quality pixel samples from previous frames and incorporate them into the computation of the current frame. This allows the renderer to render a smaller region of foveal pixels per frame by temporally reusing pixel samples that are still relevant to reconstruct a higher perceived image quality, while allowing for higher levels of foveation. Our method operates on the output of foveated rendering, and runs in under 1\,ms at 4K resolution, making it highly efficient and integrable with real-time VR and AR foveated rendering systems.

Figures (5)

• Figure 1: Visualization of Equation \ref{['eq:phat']} weights. The distribution is drawn from an approximation of visual acuity falloff given by watson2014formula, normalized by the peak retinal ganglion cell density at a 2.5$^\circ$ eccentricity from the fovea.
• Figure 2:
• Figure 3: Images and FovVideoVDP JOD maps for FOV and WRS, both using a 5$^\circ$ fovea, and undergoing the same scanpath as in Figure \ref{['fig:scanpath']}. Each image started from an initial foveation image on frame 0 (left), and progressed to their final renders (right). FovVideoVDP scores are detailed in Table \ref{['tbl:evaluation']}.
• Figure 4: WPM differences from baseline across six rendering conditions. For each fovea size, blue shows the WPM decrease under WRS conditions, and red shows the WPM decrease under FOV conditions. Slopes (s), Pearson correlations (r), and p values for the decrease are shown. WRS 10$^\circ$ and WRS 7.5$^\circ$ did not significantly decrease reading speed, while all other conditions did; in all cases, WRS reduced reading speed significantly less than FOV.
• Figure 5: Highly reflective Corridor scene with dynamic objects and camera motion and rotation, rendered with FOV + TAA (top) and WRS (bottom). In rasterization-based reflective renders, simply using motion vectors can lead to inaccurate reprojection, causing artifacts to appear when temporally accumulating samples, as pixels get reprojected to incorrect locations, impacting the reflection quality. Despite this, WRS had higher FovVideoVDP scores.