VL4Gaze: Unleashing Vision-Language Models for Gaze Following

TL;DR

VL4Gaze introduces the first large-scale gaze-understanding benchmark for vision-language models, framing gaze interpretation as a unified Gaze VQA problem with four tasks across $489K$ QA pairs and $124K$ images. It presents an automatic Gaze VQA generation pipeline with observer and gaze-target descriptions, plus a self-consistency validation and a two-stage prompting strategy to produce diverse, natural annotations. Through extensive experiments, the authors show that current VLMs struggle with gaze understanding without task-specific supervision, but fine-tuning on VL4Gaze yields substantial gains and strong cross-domain generalization, underscoring the need for targeted multi-task supervision. The dataset and code release aim to catalyze research in gaze understanding for VLMs, enabling robust, transferable gaze reasoning in real-world visual scenes.

Abstract

Human gaze provides essential cues for interpreting attention, intention, and social interaction in visual scenes, yet gaze understanding remains largely unexplored in current vision-language models (VLMs). While recent VLMs achieve strong scene-level reasoning across a range of visual tasks, there exists no benchmark that systematically evaluates or trains them for gaze interpretation, leaving open the question of whether gaze understanding can emerge from general-purpose vision-language pre-training. To address this gap, we introduce VL4Gaze, the first large-scale benchmark designed to investigate, evaluate, and unlock the potential of VLMs for gaze understanding. VL4Gaze contains 489K automatically generated question-answer pairs across 124K images and formulates gaze understanding as a unified VQA problem through four complementary tasks: (1) gaze object description, (2) gaze direction description, (3) gaze point location, and (4) ambiguous question recognition. We comprehensively evaluate both commercial and open-source VLMs under in-context learning and fine-tuning settings. The results show that even large-scale VLMs struggle to reliably infer gaze semantics and spatial localization without task-specific supervision. In contrast, training on VL4Gaze brings substantial and consistent improvements across all tasks, highlighting the importance of targeted multi-task supervision for developing gaze understanding capabilities in VLMs. We will release the dataset and code to support further research and development in this direction.

Figures (4)

• Figure 1: We introduce the first large-scale dataset VL4Gaze to explore the capabilities of VLMs for the gaze following task. Our dataset consists of 489K text-image pairs and includes four gaze understanding tasks. Building upon this dataset, we establish the first benchmark for VLM-based gaze following, unlocking the potential of VLMs and paving the way for future research in this area.
• Figure 2: Overall construction pipeline of VL4Gaze benchmark.
• Figure 3: Cross-domain generalization evaluation on the VideoAttentionTarget test set. All models are trained exclusively on the GazeFollow portion of VL4Gaze. Our Qwen3-VL-8B-Instruct model fine-tuned on VL4Gaze demonstrates superior gaze localization and reasoning ability under domain shift, clearly outperforming the non-fine-tuned Qwen3-VL-8B-Instruct baseline.
• Figure 4: Visualization of gaze-following results from three VLMs: the commercial model GPT-5, the baseline Qwen3-VL-8B-Instruct, and our fine-tuned VL4Gaze model (Ours). Green indicates the ground-truth gaze targets.