Same or Not? Enhancing Visual Perception in Vision-Language Models

TL;DR

This work tackles the gap in vision-language models' fine-grained perception by introducing Twin, a $561{,}000$ two-image instance-identity dataset that rewards subtle visual cues, and Fgvqa, a cross-domain $12{,}000$-query benchmark for evaluating fine-grained VQA. The authors post-train open-source VLMs on Twin using GRPO reinforcement learning, demonstrating substantial cross-domain gains on Fgvqa while preserving general VQA performance, and showing that dataset scale is crucial for success. They further validate the value of hard negatives (including DreamBooth-generated examples) and provide extensive ablations, including cross-domain improvements on unseen domains. The results suggest Twin as a practical drop-in augmentation for open-source VLM training, advancing perceptual precision and narrowing gaps with proprietary models, with broad implications for embodied perception and fine-grained reasoning in multimodal systems.

Abstract

Vision-language models (VLMs) excel at broad visual understanding but remain coarse-grained, exhibit visual biases, and miss subtle visual details. Existing training corpora reinforce this limitation by emphasizing general recognition ("Is it a cat or a dog?") over fine-grained perception. To address this, we introduce a new training corpus and task designed to enhance the perceptual abilities of VLMs. TWIN is a large-scale dataset of 561,000 image-pair queries that task models to determine whether two visually similar images depict the same object, encouraging attention to nuanced visual cues. The dataset spans a diverse range of everyday objects across contexts, viewpoints, and appearances. Fine-tuning VLMs on TWIN yields notable gains in fine-grained recognition, even on unseen domains such as art, animals, plants, and landmarks. To quantify these gains, we introduce FGVQA, a benchmark suite of 12,000 queries that repurposes fine-grained recognition and retrieval datasets from multiple domains. While existing VLMs struggle on FGVQA, when fine-tuned on TWIN they improve by up to 19.3%, without compromising performance on general VQA benchmarks. Finally, our TWIN dataset scales favorably with object annotations, and our analysis shows that scale is key to performance. We envision TWIN as a drop-in addition to open-source VLM training corpora, advancing perceptual precision of future models. Project webpage: https://glab-caltech.github.io/twin/

Figures (18)

• Figure 1: (a) Fine-grained visual understanding requires recognizing subtle differences between similar images. We introduce Twin, a large-scale dataset of 561K image-pair queries asking if two images depict the same object, and Fgvqa, a benchmark suite evaluating fine-grained VQA capabilities. (b) Models trained on Twin show improved fine-grained understanding on Fgvqa, even in unseen domains.
• Figure 2: (a) Twin is a large-scale VQA dataset for fine-grained visual understanding, where VLMs determine whether two images depict the same instance. Twin contains $561$K pairwise VQA queries across $1,836$ object instances, spanning $36$ categories of common objects and over $22$K images. (b) Scalability of the pairwise construction: from $1,836$ instances, Twin yields $561$K pairwise queries.
• Figure 3: Fgvqa is a suite of fine-grained VQA benchmarks spanning retail products (Twin, Iliasilias); animals and plants (Inquireinquire); Landmarkslandmarksv2; birds (Cubcub); and art (Metmet). We include two query types: pair (top row), where a VLM judges if two images depict the same instance, species, or landmark, and multi where it counts how many images match a reference.
• Figure 4: Training VLMs on Twin. We train VLMs using RL on Twin. Reward is computed by comparing the predicted answer with the ground truth pair assignment.
• Figure 5: Outputs on Fgvqa for base VLMs and variants trained on Twin. For each example, we show the source dataset, images, and both model predictions. We include both pair queries (a-d) and multi queries (e-h). Incorrect reasoning is highlighted in red and correct reasoning in green. The red circle highlights a hard-to-see object. Training on Twin enhances fine-grained grounding, leading to increased attention to part geometry and texture over the base model, even for domains that differ from the household objects in Twin.