Uncertainty-Aware Scarf Plots

TL;DR

Uncertainty in gaze-to-AOI mapping and depth ordering in AR/VR stimuli can bias eye-tracking analyses. We introduce uncertainty-aware scarf plots that extend the classic scarf plot with depth-aware segmentation and proximity-based (NN) mappings, plus a classification-confidence bar. The method formalizes three uncertainty types—position, order, and classification—and encodes them via depth scarves, NN scarves, and $p_i = w_i / \sum_j w_j$ with $w_i = 1/d_i$, while preserving depth order. The approach is demonstrated in AR using HoloLens 2 across multiple scenes and participants, showing how the plots reveal ambiguities and misclassifications at a glance, guiding interpretation and reporting.

Abstract

Multiple challenges emerge when analyzing eye-tracking data with areas of interest (AOIs) because recordings are subject to different sources of uncertainties. Previous work often presents gaze data without considering those inaccuracies in the data. To address this issue, we developed uncertainty-aware scarf plot visualizations that aim to make analysts aware of uncertainties with respect to the position-based mapping of gaze to AOIs and depth dependency in 3D scenes. Additionally, we also consider uncertainties in automatic AOI annotation. We showcase our approach in comparison to standard scarf plots in an augmented reality scenario.

Figures (8)

• Figure 1: (a) shows a True Positive (TP) classification, where the cup is classified correctly with high confidence. There can be two types of false positive (FP) classification. (b) There is no object, but a bottle gets detected, with low confidence. (c) A cup is detected as bottle, with low confidence. (Images edited with Stable Diffusion)
• Figure 2: Uncertainties in stimuli: (a) The same object (bottle) at different depths. (b) A physical object (bottle) between two virtual objects (virtual plant & virtual bust). (c) Virtual objects (virtual plant & virtual bust) at different depths. (d) A physical object (cup) at the same depth as a virtual object (virtual plant) and another virtual object (virtual bust) at a different depth. (Virtual bust from guptaarnish in (a--d): © Sketchfab Standard License: https://sketchfab.com/3d-models/marble-bust-01-4k-c9b1839068094e40bc941b047ca19a85).
• Figure 3: Each scene presents a scarf plot of each type based on recordings from a specific participant. While the standard scarf plot contains limited information, the depth scarf plot and NN scarf plot provide additional details regarding uncertainty of order and uncertainty of position within each scene. In the BB scene, the blue segments in the standard scarf plot do not reveal any insights into gaze patterns. However, the depth scarf plot detects uncertainties of order within the scene, revealing the presence of two bottles, while the NN scarf plot indicates which bottle the participant's focus is directed toward. In the VP, B & VB scene, gaze patterns caused by the task-defined visiting order can be observed in the standard scarf plot. The depth scarf plot and NN scarf plot show how the gaze ray intersects the VB and B or VP and B. The VP & VB scene highlights the detection of a non-existent AOI (Book), with the bar charts showing a low percentage of confidence for this AOI. While the standard scarf plot shows no gaze on the VB AOI due to the influence of the Book AOI, the depth scarf plot and NN scarf plot reveal the gaze on VB. In the VP, C & VB scene, we observe minimal uncertainties regarding order and position, resulting in similar-looking scarf plots. For a detailed comparison of each plot type using the same participant data, we refer to the supplemental material. (Virtual bust from guptaarnish: © Sketchfab Standard License: https://sketchfab.com/3d-models/marble-bust-01-4k-c9b1839068094e40bc941b047ca19a85)
• Figure 4: The scarf plots for the task BB. Each row (P1--P4) in the plots belongs to the data set of a participant. The plots mainly visualize the AOIs of the bottles; therefore, all plots primarily contain blue boxes for these AOIs. (Top) The standard scarf plot makes little sense here, showing only the first ray intersection. (Center) In contrast, the depth plot shows multiple segments divided into two, indicating that the participants' gaze ray intersected with the bounding box of both bottles (see (A)). (Bottom) With the NN scarf plot, we can understand when the participants were switching their gaze to another bottle (see (B) and (C)). The white spaces in the plots indicate time spans in which no gaze data was available or no hit was detected on the gaze ray or close to it.
• Figure 5: The scarf plots for the task VP, B & VB. Each row (P1--P4) in the plots belongs to the data set of a participant. (Top) The standard scarf plot shows the main AOIs present in the stimulus. (Center) The depth scarf plot provides an interpretation of the depth of the different AOIs over time. (Bottom) The NN plot is similar to the depth plot but provides insight into which AOI the participant is more likely to have looked at during specific time steps, accounting for positional uncertainties. The example shows segments where the gaze point lies on the bottle placed between the virtual plant and the virtual bust. Since the virtual objects are close to the bottle, the segment contains the color of all three AOIs as sub-segments, with the bottle and the virtual bust taking up the majority of space, since the gaze ray is the closest to those two objects. The bottle sub-segment is on the bottom since the object detection algorithm sometimes detects the bottle in front of the virtual plant. The white spaces in the plots indicate time spans in which no gaze data was available or no hit was detected on the gaze ray or close to it. (Virtual bust from guptaarnish: © Sketchfab Standard License: https://sketchfab.com/3d-models/marble-bust-01-4k-c9b1839068094e40bc941b047ca19a85)