ObjectVisA-120: Object-based Visual Attention Prediction in Interactive Street-crossing Environments

TL;DR

The paper tackles the gap between human object-based attention and computational models in interactive environments by introducing ObjectVisA-120, a VR street-crossing dataset with precise object-level annotations and ground-truth gaze data from 120 participants. It proposes oSIM, an object-focused similarity metric, and SUMGraph, a graph-enhanced attention prediction model that explicitly encodes object information via Graph VSS/C-VSS blocks and a bespoke loss that includes an oSIM term. Empirical results show that oSIM-guided training improves both object-aware and standard saliency metrics, with SUMGraph achieving state-of-the-art performance on ObjectVisA-120. The work advances safety-critical vision tasks by aligning predictions with semantically important objects and providing publicly release-ready data and code for broader adoption.

Abstract

The object-based nature of human visual attention is well-known in cognitive science, but has only played a minor role in computational visual attention models so far. This is mainly due to a lack of suitable datasets and evaluation metrics for object-based attention. To address these limitations, we present \dataset~ -- a novel 120-participant dataset of spatial street-crossing navigation in virtual reality specifically geared to object-based attention evaluations. The uniqueness of the presented dataset lies in the ethical and safety affiliated challenges that make collecting comparable data in real-world environments highly difficult. \dataset~ not only features accurate gaze data and a complete state-space representation of objects in the virtual environment, but it also offers variable scenario complexities and rich annotations, including panoptic segmentation, depth information, and vehicle keypoints. We further propose object-based similarity (oSIM) as a novel metric to evaluate the performance of object-based visual attention models, a previously unexplored performance characteristic. Our evaluations show that explicitly optimising for object-based attention not only improves oSIM performance but also leads to an improved model performance on common metrics. In addition, we present SUMGraph, a Mamba U-Net-based model, which explicitly encodes critical scene objects (vehicles) in a graph representation, leading to further performance improvements over several state-of-the-art visual attention prediction methods. The dataset, code and models will be publicly released.

Figures (4)

• Figure 1: ObjectVisA-120 dataset: (A) displays a FoV image with an overlaid visual attention map during street-crossing navigation task; (B) presents the affiliated panoptic segmentation, following the CityScapes Cordts_2016_CVPR labeling policy; (C) shows the corresponding depth map; and (D) the keypoints and edges annotations aligned with OpenPifPaf kreiss2021openpifpaf labeling policy.
• Figure 2: Comparison of oSIM with the SIM metric. Column (A) shows perfect alignment with the ground truth (SIM = oSIM = 1.0). In (B–C), predictions diverge, and both metrics decline. Unlike SIM, which drops quickly due to spatial misalignment, oSIM also accounts for object-level semantics (e.g., safety-critical approaching vehicle).
• Figure 3: Left: Overall architecture of the proposed SUMGraph model for visual attention prediction. Right: novel Graph VSS (fusion of graph and image features, in encoder) and Graph C-VSS (fusion of graph and conditional image features, in decoder) blocks for integration of additional contextual information. $\otimes$ stands for the element-wise produce operation, $\oplus$ is the element-wise addition.
• Figure 4: Qualitative visualizations of random samples. For clarity in performance improvements, both SIM and oSIM scores are provided. Column (A) presents the panoptic labels, (B) shows the ground truth attention map. The SUMGraph model (D) demonstrates improved performance compared to the baseline, with the incorporation of oSIM (columns C vs. D, and E vs. F) leading to consistent improvements.