VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues

TL;DR

This work introduces VLM2-Bench, a comprehensive benchmark designed to probe vision-language models’ ability to visually link matching cues across images and videos. It organizes cues into General, Object-centric, and Person-centric categories, encompassing 9 subtasks and over 3,000 QA pairs evaluated across twelve VLMs with varied prompt strategies. The study reveals a substantial performance gap between humans and current models, highlights consistent error patterns in general cue tasks, and shows relative strength for person-centric cue linking, prompting calls for stronger visual foundations, clearer integration of language-based reasoning, and next-generation vision-text training that emphasizes independent visual cue reasoning. The findings advance understanding of multimodal reasoning bottlenecks and offer concrete guidance for improving visual cue linking in real-world applications.

Abstract

Visually linking matching cues is a crucial ability in daily life, such as identifying the same person in multiple photos based on their cues, even without knowing who they are. Despite the extensive knowledge that vision-language models (VLMs) possess, it remains largely unexplored whether they are capable of performing this fundamental task. To address this, we introduce \textbf{VLM2-Bench}, a benchmark designed to assess whether VLMs can Visually Link Matching cues, with 9 subtasks and over 3,000 test cases. Comprehensive evaluation across twelve VLMs, along with further analysis of various language-side and vision-side prompting methods, leads to a total of eight key findings. We identify critical challenges in models' ability to link visual cues, highlighting a significant performance gap. Based on these insights, we advocate for (i) enhancing core visual capabilities to improve adaptability and reduce reliance on prior knowledge, (ii) establishing clearer principles for integrating language-based reasoning in vision-centric tasks to prevent unnecessary biases, and (iii) shifting vision-text training paradigms toward fostering models' ability to independently structure and infer relationships among visual cues.

Figures (19)

• Figure 1: Previous benchmarks fail to assess the ability to link matching visual cues, whereas our VLM2-Bench explicitly tests this ability, as shown in the example where the model need to identify the reappearance of the same person by linking visual cues, like facial features or clothing, across non-adjacent frames.
• Figure 2: Overview of VLM2-Bench. The benchmark is categorized into three subsets based on visual cues: GC (General Cue), OC (Object-centric Cue), and PC (Person-centric Cue), each comprising multiple subtasks. To comprehensively evaluate VLMs' ability to visually link matching cues, the benchmark includes diverse question formats—T/F , multiple-choice , numerical , and open-ended —ensuring a comprehensive evaluation.
• Figure 3: Construction of GC: (i) We start by manually verifying the edited image data based on three key criteria. (ii) A VLM is then prompted to generate captions for each image, followed by salient score-based filtering to retain the challenging cases. (iii) Finally, visual cues are extracted from two sources and incorporated into a QA prompt, guiding an LLM to generate both positive and negative answer pairs.
• Figure 4: Statistical overview of VLM2-Bench. The pie chart shows the distribution of 9 subtasks across the 3 main categories of visual cues. The bar plot illustrates the percentage breakdown by question format.
• Figure 5: Evaluation results on VLM2-Bench, covering Mat (Matching), Trk (Tracking), Cpr (Comparison), Cnt (Counting), Grp (Grouping), and VID (Video Identity Describing). The highest, second, and third highest scores are highlighted. *: Overall excludes the VID due to the lack of a chance-level baseline for open-ended tasks.