A Robotics-Inspired Scanpath Model Reveals the Importance of Uncertainty and Semantic Object Cues for Gaze Guidance in Dynamic Scenes

TL;DR

This paper tackles gaze guidance in dynamic scenes by proposing a robotics-inspired, image-computable framework that tightly couples interdependent object segmentation and saccadic decision-making. It uses a Bayesian particle filter to maintain an uncertainty-quantified segmentation and a drift-diffusion model to drive saccades, with scene relevance from semantic cues and gaze-dependent sensitivity shaping the evidence for each potential target. The authors show that uncertainty-driven exploration and semantic-object cues are pivotal for producing human-like scanpaths, including temporal dynamics akin to inhibition of return, without explicitly implementing IOR. The modular AICON-based design enables principled ablation studies and extensions (e.g., saccadic momentum, pre-saccadic attention), offering a versatile platform for testing attentional hypotheses in naturalistic, dynamic vision tasks.

Abstract

The objects we perceive guide our eye movements when observing real-world dynamic scenes. Yet, gaze shifts and selective attention are critical for perceiving details and refining object boundaries. Object segmentation and gaze behavior are, however, typically treated as two independent processes. Here, we present a computational model that simulates these processes in an interconnected manner and allows for hypothesis-driven investigations of distinct attentional mechanisms. Drawing on an information processing pattern from robotics, we use a Bayesian filter to recursively segment the scene, which also provides an uncertainty estimate for the object boundaries that we use to guide active scene exploration. We demonstrate that this model closely resembles observers' free viewing behavior on a dataset of dynamic real-world scenes, measured by scanpath statistics, including foveation duration and saccade amplitude distributions used for parameter fitting and higher-level statistics not used for fitting. These include how object detections, inspections, and returns are balanced and a delay of returning saccades without an explicit implementation of such temporal inhibition of return. Extensive simulations and ablation studies show that uncertainty promotes balanced exploration and that semantic object cues are crucial to forming the perceptual units used in object-based attention. Moreover, we show how our model's modular design allows for extensions, such as incorporating saccadic momentum or pre-saccadic attention, to further align its output with human scanpaths.

Figures (15)

• Figure 1: Saccadic decisions and object perception influence each other, as reflected by their interconnection in our model. We illustrate the information flow in our model during the processing of a single frame from a dynamic video. Object segmentation is informed by multiple global object cues and a high-confidence prompted segmentation of the foveated object. The segmented objects act as perceptual units for the saccade target selection. The uncertainty over object segmentation plays a key role in driving exploration while being resolved through high-confidence measurements at the current gaze position. As both the dynamic scene and gaze change over time, the recursive estimator continuously updates the segmentation and its uncertainty.
• Figure 2: Our model combines multiple object cues to estimate both object segmentation and its uncertainty recursively. We integrate foveated and global segmentations of the scene (left) in a Bayesian filter (middle), which maintains a belief over the current state, represented by a weighted set of multiple possible segmentation samples ($14$ example samples from the full set of $50$ are shown). We then compute the currently most likely segmentation and its uncertainty (right), which we use to inform saccadic decisions.
• Figure 3: Our model makes saccadic decisions based on objects and is driven by uncertainty. It combines the uncertainty over object segmentation with salience and gaze-dependent sensitivity (left) into evidence for individual objects (middle). This evidence is then accumulated for each object in a drift-diffusion process (right). As soon as its threshold is passed, a saccade to this object is executed, otherwise the gaze smoothly pursues the currently foveated object.
• Figure 4: The predicted scanpaths of our model show human-like exploration in dynamic scenes. In this video of the test dataset, the model first follows uncertainty and detects two novel objects (dancers) (a), then returns to the first before detecting another one (b), which is then further inspected primarily due to its high visual saliency (c and d). For a video version, see \ref{['app:videos']}.
• Figure 5: Aggregated statistics of the simulated scanpaths of the base model resemble the human eye-tracking data. (a) Histogram of the duration of all foveations in the human ground truth data (red) and the base model (blue). (b) Histogram of the saccade amplitude distributions. (c) Percentage of foveation events in the categories “Background” (maroon), “Detection” (orange), “Inspection” (yellow), and “Return” (khaki) across all human (solid) and model (dashed) scanpaths as a function of time. (d) Median duration of the preceding foveation durations for each saccade. We applied a centered circular moving average across 5 bins (12$^{\circ}$ bin size) to reduce fluctuations in the median.