Look Hear: Gaze Prediction for Speech-directed Human Attention

TL;DR

This work tackles the challenge of predicting human gaze incrementally as a referring expression is heard while viewing a scene. It introduces ART, a multimodal transformer that jointly learns gaze guidance and object grounding, emitting packs of fixations per word via an autoregressive decoder, and trains on partial expressions through auxiliary grounding objectives. To support this, the large RefCOCO-Gaze dataset collects 19,738 gaze scanpaths from 220 participants for 2,094 image-expression pairs, enabling evaluation of dynamic, word-level gaze alignment. ART outperforms baselines across multiple spatiotemporal metrics, and ablations show that pretraining on grounding tasks and incorporating both localization and target-category losses are crucial, with qualitative results revealing waiting, scanning, and verification strategies akin to human attention. The framework generalizes to categorical search and has practical implications for time-sensitive voice-guided HCI in AR/VR and related domains, where predicting gaze can improve efficiency and user experience.

Abstract

For computer systems to effectively interact with humans using spoken language, they need to understand how the words being generated affect the users' moment-by-moment attention. Our study focuses on the incremental prediction of attention as a person is seeing an image and hearing a referring expression defining the object in the scene that should be fixated by gaze. To predict the gaze scanpaths in this incremental object referral task, we developed the Attention in Referral Transformer model or ART, which predicts the human fixations spurred by each word in a referring expression. ART uses a multimodal transformer encoder to jointly learn gaze behavior and its underlying grounding tasks, and an autoregressive transformer decoder to predict, for each word, a variable number of fixations based on fixation history. To train ART, we created RefCOCO-Gaze, a large-scale dataset of 19,738 human gaze scanpaths, corresponding to 2,094 unique image-expression pairs, from 220 participants performing our referral task. In our quantitative and qualitative analyses, ART not only outperforms existing methods in scanpath prediction, but also appears to capture several human attention patterns, such as waiting, scanning, and verification.

Figures (9)

• Figure 1: RefCOCO-Gaze Dataset. Sample image-expression pairs and corresponding scanpaths under our incremental object referral task. Fixations (denoted by circles numbered with fixation order) are color-coded to the corresponding word in the referring expression (above each image). Fixations color-coded to [BOT] occurred before the expression started, and fixations color-coded to [EOT] occurred after the expression ended. Blue bounding boxes indicating referred objects were not visible during trials.
• Figure 2: Behavioral data collection in our incremental object referral paradigm.
• Figure 3: Attention in Referral Transformer (ART) Architecture. On each pass after comprehending a new word, the model takes an image $I$ and tokens $T_j$ of prefix $R_j$ of the referring expression as input and generates a possibly empty sub-sequence of fixations based on previous fixation history encoded by a fixation context encoder.
• Figure 4: Qualitative results. Scanpaths from humans and three scanpath prediction models on three trials exhibiting strategic fixation behavior. Fixations (denoted by circles numbered with fixation order) are color-coded to corresponding words in the referring expression (above each row). Fixations color-coded to [BOT] occurred before the expression started, and those color-coded to [EOT] occurred after the expression ended. Blue bounding boxes indicating the referred objects are not visible during trials. Our model generates the most human-like scanpaths for incremental object referral.
• Figure 5: Excluded Samples