Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Zero-Shot Capabilities of Vision-Language Models for Improving Gaze Following

Anshul Gupta, Pierre Vuillecard, Arya Farkhondeh, Jean-Marc Odobez

TL;DR

The paper addresses gaze following in complex scenes by leveraging zero-shot cues from Vision-Language Models to capture a broad range of contextual cues. It systematically evaluates multiple VLMs, visual and text prompting strategies, and in-context learning, identifying BLIP-2 as a strong zero-shot cue extractor and demonstrating that ensembling prompts improves robustness. The authors integrate these cues into a transformer-based gaze-following model, showing improved generalization, especially when using larger cue sets and an early fusion scheme. The work suggests zero-shot VLM cues offer scalable, domain-adaptive benefits for gaze following, with potential extensions to HOI-derived cues and further prompting strategies.

Abstract

Contextual cues related to a person's pose and interactions with objects and other people in the scene can provide valuable information for gaze following. While existing methods have focused on dedicated cue extraction methods, in this work we investigate the zero-shot capabilities of Vision-Language Models (VLMs) for extracting a wide array of contextual cues to improve gaze following performance. We first evaluate various VLMs, prompting strategies, and in-context learning (ICL) techniques for zero-shot cue recognition performance. We then use these insights to extract contextual cues for gaze following, and investigate their impact when incorporated into a state of the art model for the task. Our analysis indicates that BLIP-2 is the overall top performing VLM and that ICL can improve performance. We also observe that VLMs are sensitive to the choice of the text prompt although ensembling over multiple text prompts can provide more robust performance. Additionally, we discover that using the entire image along with an ellipse drawn around the target person is the most effective strategy for visual prompting. For gaze following, incorporating the extracted cues results in better generalization performance, especially when considering a larger set of cues, highlighting the potential of this approach.

Exploring the Zero-Shot Capabilities of Vision-Language Models for Improving Gaze Following

TL;DR

The paper addresses gaze following in complex scenes by leveraging zero-shot cues from Vision-Language Models to capture a broad range of contextual cues. It systematically evaluates multiple VLMs, visual and text prompting strategies, and in-context learning, identifying BLIP-2 as a strong zero-shot cue extractor and demonstrating that ensembling prompts improves robustness. The authors integrate these cues into a transformer-based gaze-following model, showing improved generalization, especially when using larger cue sets and an early fusion scheme. The work suggests zero-shot VLM cues offer scalable, domain-adaptive benefits for gaze following, with potential extensions to HOI-derived cues and further prompting strategies.

Abstract

Contextual cues related to a person's pose and interactions with objects and other people in the scene can provide valuable information for gaze following. While existing methods have focused on dedicated cue extraction methods, in this work we investigate the zero-shot capabilities of Vision-Language Models (VLMs) for extracting a wide array of contextual cues to improve gaze following performance. We first evaluate various VLMs, prompting strategies, and in-context learning (ICL) techniques for zero-shot cue recognition performance. We then use these insights to extract contextual cues for gaze following, and investigate their impact when incorporated into a state of the art model for the task. Our analysis indicates that BLIP-2 is the overall top performing VLM and that ICL can improve performance. We also observe that VLMs are sensitive to the choice of the text prompt although ensembling over multiple text prompts can provide more robust performance. Additionally, we discover that using the entire image along with an ellipse drawn around the target person is the most effective strategy for visual prompting. For gaze following, incorporating the extracted cues results in better generalization performance, especially when considering a larger set of cues, highlighting the potential of this approach.
Paper Structure (18 sections, 3 equations, 11 figures, 5 tables)

This paper contains 18 sections, 3 equations, 11 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related-Work
  3. Contextual Cues Extraction
  4. Method
  5. Experiments
  6. ITM Results
  7. VQA Results
  8. Text-Improved Gaze Following
  9. Method
  10. Experiments
  11. Results
  12. Discussion
  13. Conclusion
  14. Appendix
  15. Example of visual prompts
  16. ...and 3 more sections

Figures (11)

  • Figure 1: As humans, we rely on various sources of information to predict a person's gaze target. This image shows contextual information that could be valuable.
  • Figure 2: Results of different visual prompting approach on Childplay. Image corresponds to input the full image whereas person refers to the use of person crop as input.
  • Figure 3: Results of different VLMs following the ITM approach on AVA. Three VLMs are compared across different classes categorized as Pose (P), Person-Person Interaction (P-P), and Person-Object Interaction (P-O).
  • Figure 4: Results of different templates using BLIP-2 on AVA. Six templates are compared across different classes.
  • Figure 5: Results of BLIP-2 vqa with and without in-context learning, vqa ICL and vqa respectively, on ChildPlay. It is compared with the VLMs CLIP, BLIP and BLIP-2.
  • ...and 6 more figures