Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UOUO: Uncontextualized Uncommon Objects for Measuring Knowledge Horizons of Vision Language Models

Xinyu Pi, Mingyuan Wu, Jize Jiang, Haozhen Zheng, Beitong Tian, Chengxiang Zhai, Klara Nahrstedt, Zhiting Hu

TL;DR

The paper tackles the problem of assessing Vision-Language Models on long-tail, uncommon objects by introducing the UOUO benchmark, a million-scale dataset of uncontextualized rare objects and a scalable data-curation/cleaning pipeline. It systematically compares large- and small-scale VLMs, finding that smaller models struggle significantly with uncommon objects, while larger models (including GPT-4 variants) handle these challenges much better. The methodology combines domain-focused data collection (manufacturing), CLIP-based automatic filtering, and MMD-driven hard-instance generation with Mosaic augmentations to create robust test samples. This work highlights the importance of long-tail evaluation for robust multimodal understanding and provides an extensible framework for future exploration across domains and modalities.

Abstract

Smaller-scale Vision-Langauge Models (VLMs) often claim to perform on par with larger models in general-domain visual grounding and question-answering benchmarks while offering advantages in computational efficiency and storage. However, their ability to handle rare objects, which fall into the long tail of data distributions, is less understood. To rigorously evaluate this aspect, we introduce the "Uncontextualized Uncommon Objects" (UOUO) benchmark. This benchmark focuses on systematically testing VLMs with both large and small parameter counts on rare and specialized objects. Our comprehensive analysis reveals that while smaller VLMs maintain competitive performance on common datasets, they significantly underperform on tasks involving uncommon objects. We also propose an advanced, scalable pipeline for data collection and cleaning, ensuring the UOUO benchmark provides high-quality, challenging instances. These findings highlight the need to consider long-tail distributions when assessing the true capabilities of VLMs.

UOUO: Uncontextualized Uncommon Objects for Measuring Knowledge Horizons of Vision Language Models

TL;DR

The paper tackles the problem of assessing Vision-Language Models on long-tail, uncommon objects by introducing the UOUO benchmark, a million-scale dataset of uncontextualized rare objects and a scalable data-curation/cleaning pipeline. It systematically compares large- and small-scale VLMs, finding that smaller models struggle significantly with uncommon objects, while larger models (including GPT-4 variants) handle these challenges much better. The methodology combines domain-focused data collection (manufacturing), CLIP-based automatic filtering, and MMD-driven hard-instance generation with Mosaic augmentations to create robust test samples. This work highlights the importance of long-tail evaluation for robust multimodal understanding and provides an extensible framework for future exploration across domains and modalities.

Abstract

Smaller-scale Vision-Langauge Models (VLMs) often claim to perform on par with larger models in general-domain visual grounding and question-answering benchmarks while offering advantages in computational efficiency and storage. However, their ability to handle rare objects, which fall into the long tail of data distributions, is less understood. To rigorously evaluate this aspect, we introduce the "Uncontextualized Uncommon Objects" (UOUO) benchmark. This benchmark focuses on systematically testing VLMs with both large and small parameter counts on rare and specialized objects. Our comprehensive analysis reveals that while smaller VLMs maintain competitive performance on common datasets, they significantly underperform on tasks involving uncommon objects. We also propose an advanced, scalable pipeline for data collection and cleaning, ensuring the UOUO benchmark provides high-quality, challenging instances. These findings highlight the need to consider long-tail distributions when assessing the true capabilities of VLMs.
Paper Structure (16 sections, 2 equations, 3 figures, 2 tables)

This paper contains 16 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Real-world VQA Benchmarks
  4. Existing Datasets with Uncommon Object Labels
  5. Data Curation and Filtering
  6. Domain Selection and Scraping
  7. Mannual Annotation
  8. Automatic Data Cleaning
  9. Test Instances Generation
  10. Background Removal and Decontextualization
  11. Testing Instances Generation
  12. Experiment
  13. Procedures.
  14. Observations and Analysis.
  15. Conclusion
  16. ...and 1 more sections

Figures (3)

  • Figure 1: UOUO Data Curation Pipeline. Snowflake means frozen weights, and fire means tune-able weights.
  • Figure 2: With MMD, we can retrieve harder negative examples and construct higher-quality test instanecs.
  • Figure 3: Wikipedia Industry List