Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning

TL;DR

This work investigates whether Vision-Language Models can serve as effective guessers for when and where an image was taken. It introduces WikiTiLo, a socio-culturally rich image dataset, and a two-stage probing framework (Recognition for cue extraction and Reasoning for open-ended inference) evaluated on discriminative VLMs (e.g., CLIP, BLIP) and generative VLMs (e.g., OpenFlamingo, LLaMA-Adapter). Results show discriminative encoders retain informative, context-agnostic features enabling times and location recognition, while generative models struggle to ground these cues in reasoning tasks, even with in-context demonstrations. The findings underscore the need to improve grounding in multimodal models and provide a valuable benchmark and baselines for future work in commonsense, times, and location reasoning with VLMs. The work also contributes to reproducibility by releasing WikiTiLo and associated code.

Abstract

Vision-Language Models (VLMs) are expected to be capable of reasoning with commonsense knowledge as human beings. One example is that humans can reason where and when an image is taken based on their knowledge. This makes us wonder if, based on visual cues, Vision-Language Models that are pre-trained with large-scale image-text resources can achieve and even outperform human's capability in reasoning times and location. To address this question, we propose a two-stage \recognition\space and \reasoning\space probing task, applied to discriminative and generative VLMs to uncover whether VLMs can recognize times and location-relevant features and further reason about it. To facilitate the investigation, we introduce WikiTiLo, a well-curated image dataset compromising images with rich socio-cultural cues. In the extensive experimental studies, we find that although VLMs can effectively retain relevant features in visual encoders, they still fail to make perfect reasoning. We will release our dataset and codes to facilitate future studies.

Figures (9)

• Figure 1: An example image of times and location reasoning in WikiTiLo: Can you tell where and when is this picture taken?
• Figure 2: We apply a two-stage probing task Recognition and Reasoning to both discriminative and generative VLMs. We evaluate Recognition on discriminative VLMs and Reasoning on generative VLMs with the visual encoder intact.
• Figure 3: Some example images in the WikiTiLo exhibit abundant visual cues with a sociocultural background, such as stylish buildings, text on images, and traditional clothing. These visual cues enable humans to reason and draw conclusions based on the provided evidence.
• Figure 4: In Recognition, we calculate discriminative VLMs' posterior $\mathbb{P}(a|v)$ of the context-agnostic visual features $v$ of an image $i$. In Reasoning, the context-conditioned visual features $v_c$ are dependent on intact visual features $v$ and question context $c$, including instructions, prompts, and demonstrations; the posterior probability of possible answers is denoted as $\mathbb{P}(a|v_c)$.
• Figure 5: We list templates for each protocol for Reasoning$_{\textsc{Location}}$. For OpenFlamingo, few-shot examples are used to facilitate in-context learning. In the case of the zero-shot setting, two text samples are retained but without image tokens <image>.