Understanding and Modeling the Effects of Task and Context on Drivers' Gaze Allocation

TL;DR

The paper tackles driver gaze prediction by integrating bottom-up saliency with explicit task and context information. ItCorrects the DR(eye)VE data pipeline to reduce noise, adds per-frame task/context annotations, and benchmarks multiple baselines, introducing SCOUT, a Transformer-based model that modulates gaze with action and context signals. Key findings show that data cleaning improves all models and that task/context modulation yields substantial gains, especially in safety-critical maneuvers and complex intersections. This work enhances driver monitoring and assistance systems by enabling context-aware gaze prediction and provides extended data and code to support future research.

Abstract

To further advance driver monitoring and assistance systems, it is important to understand how drivers allocate their attention, in other words, where do they tend to look and why. Traditionally, factors affecting human visual attention have been divided into bottom-up (involuntary attraction to salient regions) and top-down (driven by the demands of the task being performed). Although both play a role in directing drivers' gaze, most of the existing models for drivers' gaze prediction apply techniques developed for bottom-up saliency and do not consider influences of the drivers' actions explicitly. Likewise, common driving attention benchmarks lack relevant annotations for drivers' actions and the context in which they are performed. Therefore, to enable analysis and modeling of these factors for drivers' gaze prediction, we propose the following: 1) we correct the data processing pipeline used in DR(eye)VE to reduce noise in the recorded gaze data; 2) we then add per-frame labels for driving task and context; 3) we benchmark a number of baseline and SOTA models for saliency and driver gaze prediction and use new annotations to analyze how their performance changes in scenarios involving different tasks; and, lastly, 4) we develop a novel model that modulates drivers' gaze prediction with explicit action and context information. While reducing noise in the DR(eye)VE gaze data improves results of all models, we show that using task information in our proposed model boosts performance even further compared to bottom-up models on the cleaned up data, both overall (by 24% KLD and 89% NSS) and on scenarios that involve performing safety-critical maneuvers and crossing intersections (by up to 10--30% KLD). Extended annotations and code are available at https://github.com/ykotseruba/SCOUT.

Figures (7)

• Figure 1: Example of misalignment by 9 frames. The car in the driver's view (ETG) is to the left of the ego-vehicle but is to the right in the rooftop camera view (GARorig). Our recomputed GARnew shows a correct alignment.
• Figure 2: Typical sources of noise in the original DR(eye)VE ground truth and effects of the proposed corrections. ETG and GAR refer to driver's and rooftop camera views, respectively. Fixations (red circles) are shown for time $t$. To form the ground truth (GT), fixations are aggregated over 1s (or $t\pm12$). To compensate for motion in the frame, the original ground truth (GT$_{\mathrm{orig}}$) uses homography transformation and the new one (GT$_{\mathrm{new}}$) relies on optical flow, resulting in less noisy saliency maps. Top row: The vertical "trail" of points in GT$_{\mathrm{orig}}$ corresponds to a saccade towards the speedometer and is absent in GT$_{\mathrm{new}}$ after removal of saccades and in-vehicle fixations. Bottom row: Incorrect mapping of driver's gaze outside the GAR view results in a random pattern in GT$_{\mathrm{orig}}$. In GT$_{\mathrm{new}}$, such out-of-frame fixations are pushed towards the edge of the frame to preserve the direction and elevation of the driver's gaze. Additional examples are provided in Appendix \ref{['suppl:gt']}.
• Figure 3: Architecture of SCOUT model: two encoders for visual and task/context data, multi-head attention (MHA) modulation blocks, and a decoder. Text labels $l_1, l_2, \dots l_n$ and numeric features $n_1, n_2, \dots, n_m$ represent task and context information. Dashed arrows indicate possible connections between modules depending on how many modulation blocks are used for fusing visual and task/context features.
• Figure 4: Examples of yielding scenarios at different intersections types while performing a turn. The ground truth (in white) is overlaid on top of the images in the first column.
• Figure 5: Common scenarios where driver's gaze (shown as a red circle) is incorrectly projected onto the scene. Left column shows the view from the driver's eye-tracking glasses (ETG) and the right column shows the view of the rooftop Garmin camera (GAR). The following scenarios are demonstrated: a) looking down; b) checking rearview mirror; c) looking outside the GAR camera view during turns; d) raindrops on the windshield.