GPU accelerated surface-based gaze mapping for XR experiences

TL;DR

This work tackles the difficulty of mapping gaze data to 6DoF XR scenes by introducing a GPU-accelerated, surface-based fixation density mapping pipeline that operates independently of mesh resolution and UV mappings. It uses a quasi-uniform surface sampling on triangles, a Gaussian projection model for gaze with occlusion handling, and normalization, all implemented as parallel GPU kernels with a filtering optimization for real-time performance. The approach yields accurate, occlusion-aware heatmaps that can be exported for external analysis and integrated into XR toolchains like PLUME, demonstrated on challenging scenes with large mesh counts and high sampling density. The combination of interactive generation, robust surface grounding, and open-source release enables scalable 6DoF gaze datasets and practical ground-truth for attention prediction and rendering optimizations in XR.

Abstract

Extended reality is a fast-growing domain for which there is an increasing need to analyze and understand user behavior. In particular, understanding human visual attention during immersive experiences is crucial for many applications. The visualization and analysis of visual attention are commonly done by building fixation density maps from eye-tracking data. Such visual attention mapping is well mastered for 3 degrees of freedom (3DoF) experiences (\textit{i.e.}, involving 360 images or videos) but much less so for 6DoFs data, when the user can move freely in the 3D space. In that case, the visual attention information has to be mapped onto the 3D objects themselves. Some solutions exist for constructing such surface-based 6DoFs attention maps, however, they own several drawbacks: processing time, strong dependence on mesh resolution and/or texture mapping, and/or unpractical data representation for further processing. In this context, we propose a novel GPU-based algorithm that resolves the issues above while being generated in interactive time and rendered in real-time. Experiment on a challenging scene demonstrates the accuracy and robustness of our approach. To stimulate research in this area, the source code is publicly released and integrated into PLUME for ease of use in XR experiments.

Figures (10)

• Figure 1: Texel based 3D attention map and its associated texture image. (Reprinted from Pfeiffer et MemiliPfeiffer_Memili_2016)
• Figure 2: Uniformity of samples distribution with and without adaptive resolution. The edges of the triangles are represented in red, the samples are represented as black dots. In the left image, samples are further apart from each other in the stretched triangle, this will be even more noticeable in models with larger triangles for low detailed areas.
• Figure 3: Gaussian projection principle. $\vb{r}$ is the gaze ray direction, $\vb{p}$ is a vector going from the camera to a considered sample point. The cone represents the deviation of the gaze ray with $\sigma$ corresponding to the angular deviation $\theta$. Note that the Gaussian does also propagate outside of the cone represented on this figure. In our case, we propagate the values up to $4\sigma$, corresponding to approximately 100% of the Gaussian distribution.
• Figure 4: The blue sample point on the front of the model is closer to the camera than the green one in the back. In the z-buffer (right image), the depth of the closest point (blue) is stored in the pixel corresponding to its projection in screen space. The smaller the depth value, the whiter the pixel. When performing the occlusion check for the green sample, its depth is different from the one stored inside its corresponding pixel inside the z-buffer ($\pm\epsilon$), meaning that the green point is not visible. An example of multiple fixations projected onto the mesh with occlusion support can be seen on the right image.
• Figure 5: The accumulation kernel can be run separately for each sample. By using the GPU, this allows for a fast generation of the fixation density map. The same is done for the normalization kernel.