Cross-Paradigm Evaluation of Gaze-Based Semantic Object Identification for Intelligent Vehicles

TL;DR

This work tackles gaze-based semantic object identification in driving scenes by performing a cross-paradigm evaluation across object-detection, segmentation-assisted, and Vision-Language Model (VLM) approaches. A new benchmark derived from the BDD100K dataset, with four environmental conditions and five target classes, enables a controlled comparison of methods including YOLOv13, SAM2-based pipelines, and Qwen2.5-VL-7b/32b. Key findings show that direct object detection and large VLMs offer the strongest performance, while segmentation-based pipelines suffer from a part-versus-whole semantic gap and poor recall; large VLMs offer robustness in low-light and small-object scenarios at the cost of computation. The results provide practical guidance for designing human-aware driver monitoring systems, highlighting a trade-off between real-time deployment (favoring YOLOv13) and robustness to challenging conditions (favoring Qwen2.5-VL-32b), and point to future work integrating temporal information and hardware-aware optimizations.

Abstract

Understanding where drivers direct their visual attention during driving, as characterized by gaze behavior, is critical for developing next-generation advanced driver-assistance systems and improving road safety. This paper tackles this challenge as a semantic identification task from the road scenes captured by a vehicle's front-view camera. Specifically, the collocation of gaze points with object semantics is investigated using three distinct vision-based approaches: direct object detection (YOLOv13), segmentation-assisted classification (SAM2 paired with EfficientNetV2 versus YOLOv13), and query-based Vision-Language Models, VLMs (Qwen2.5-VL-7b versus Qwen2.5-VL-32b). The results demonstrate that the direct object detection (YOLOv13) and Qwen2.5-VL-32b significantly outperform other approaches, achieving Macro F1-Scores over 0.84. The large VLM (Qwen2.5-VL-32b), in particular, exhibited superior robustness and performance for identifying small, safety-critical objects such as traffic lights, especially in adverse nighttime conditions. Conversely, the segmentation-assisted paradigm suffers from a "part-versus-whole" semantic gap that led to large failure in recall. The results reveal a fundamental trade-off between the real-time efficiency of traditional detectors and the richer contextual understanding and robustness offered by large VLMs. These findings provide critical insights and practical guidance for the design of future human-aware intelligent driver monitoring systems.

Figures (15)

• Figure 1: Representative examples from the four annotated scenarios and annotation protocol.
• Figure 2: Distribution of annotated instances per class.
• Figure 3: Distribution and density of instances across scenarios.
• Figure 4: Visualization of the three methodological paradigms.
• Figure 5: The structured prompt used for the VLM based methods.