Caption-Driven Explorations: Aligning Image and Text Embeddings through Human-Inspired Foveated Vision

TL;DR

This work tackles task-driven visual attention during image captioning by introducing CapMIT1003, a dataset of click-contingent explorations paired with captions, and NevaClip, a zero-shot method that aligns foveated image embeddings with caption embeddings. NevaClip optimizes the cosine similarity loss $S(e^\pi,e^C)$ between foveated image embeddings $e^\pi$ and caption embeddings $e^C$, iteratively refining fixations to minimize alignment error. Empirical results on CapMIT1003 show NevaClip with the correct caption yields high scanpath plausibility and outperforms baselines; results on MIT1003 indicate overlapping patterns between caption-driven and free-viewing attention. The work provides a public dataset and a zero-shot scanpath predictor that advance understanding of attention dynamics in vision-and-language tasks.

Abstract

Understanding human attention is crucial for vision science and AI. While many models exist for free-viewing, less is known about task-driven image exploration. To address this, we introduce CapMIT1003, a dataset with captions and click-contingent image explorations, to study human attention during the captioning task. We also present NevaClip, a zero-shot method for predicting visual scanpaths by combining CLIP models with NeVA algorithms. NevaClip generates fixations to align the representations of foveated visual stimuli and captions. The simulated scanpaths outperform existing human attention models in plausibility for captioning and free-viewing tasks. This research enhances the understanding of human attention and advances scanpath prediction models.

Figures (1)

• Figure 1: Method Overview (NevaClip). The third fixation $\xi_{i,3}$ is optimized in order to maximise the alignment between the caption representation $e^C$ and the foveated image representation $e^\pi$ under the current fixation point.