Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WorldVQA: Measuring Atomic World Knowledge in Multimodal Large Language Models

Runjie Zhou, Youbo Shao, Haoyu Lu, Bowei Xing, Tongtong Bai, Yujie Chen, Jie Zhao, Lin Sui, Haotian Yao, Zijia Zhao, Hao Yang, Haoning Wu, Zaida Zhou, Jinguo Zhu, Zhiqi Huang, Yiping Bao, Yangyang Liu, Y. Charles, Xinyu Zhou

TL;DR

WorldVQA presents a targeted benchmark to isolate and measure atomic visual knowledge in Multimodal LLMs, separating visual grounding from reasoning to quantify factual recall and hallucination. The authors implement a rigorous data collection and verification pipeline, including a four-pronged design principle, a three-stage data curation process, and a model-based difficulty stratification with dual automated and human verification. Experiments across frontier and open-source models reveal substantial gaps in atomic visual grounding, especially in Nature and Culture domains, and expose persistent calibration issues where models are overconfident notwithstanding accuracy. The benchmark provides a standardized, densely annotated resource to evaluate visual factuality and guides future data collection, model alignment, and calibration improvements in multimodal systems.

Abstract

We introduce WorldVQA, a benchmark designed to evaluate the atomic visual world knowledge of Multimodal Large Language Models (MLLMs). Unlike current evaluations, which often conflate visual knowledge retrieval with reasoning, WorldVQA decouples these capabilities to strictly measure "what the model memorizes." The benchmark assesses the atomic capability of grounding and naming visual entities across a stratified taxonomy, spanning from common head-class objects to long-tail rarities. We expect WorldVQA to serve as a rigorous test for visual factuality, thereby establishing a standard for assessing the encyclopedic breadth and hallucination rates of current and next-generation frontier models.

WorldVQA: Measuring Atomic World Knowledge in Multimodal Large Language Models

TL;DR

WorldVQA presents a targeted benchmark to isolate and measure atomic visual knowledge in Multimodal LLMs, separating visual grounding from reasoning to quantify factual recall and hallucination. The authors implement a rigorous data collection and verification pipeline, including a four-pronged design principle, a three-stage data curation process, and a model-based difficulty stratification with dual automated and human verification. Experiments across frontier and open-source models reveal substantial gaps in atomic visual grounding, especially in Nature and Culture domains, and expose persistent calibration issues where models are overconfident notwithstanding accuracy. The benchmark provides a standardized, densely annotated resource to evaluate visual factuality and guides future data collection, model alignment, and calibration improvements in multimodal systems.

Abstract

We introduce WorldVQA, a benchmark designed to evaluate the atomic visual world knowledge of Multimodal Large Language Models (MLLMs). Unlike current evaluations, which often conflate visual knowledge retrieval with reasoning, WorldVQA decouples these capabilities to strictly measure "what the model memorizes." The benchmark assesses the atomic capability of grounding and naming visual entities across a stratified taxonomy, spanning from common head-class objects to long-tail rarities. We expect WorldVQA to serve as a rigorous test for visual factuality, thereby establishing a standard for assessing the encyclopedic breadth and hallucination rates of current and next-generation frontier models.
Paper Structure (25 sections, 5 figures, 3 tables)

This paper contains 25 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Data Collection and Verification
  3. Design Principles and Criteria
  4. Data Quality
  5. Data Curation Pipeline
  6. Model-Based Difficulty Stratification
  7. Dual-Verification Mechanism
  8. Grading and Metrics
  9. Benchmark Statistics
  10. Experiments
  11. Settings
  12. Main Results
  13. Validation of Difficulty Stratification
  14. Calibration Analysis
  15. Related Work and Discussion
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Overall Model Accuracy on WorldVQA. While the Gemini-3-pro (47.4%) and Kimi K2.5 (46.3%) currently lead the field, no evaluated model surpasses the 50% accuracy threshold, underscoring the significant challenge of grounding atomic visual knowledge.
  • Figure 2: A visual overview of the WorldVQA dataset. The benchmark is organized into nine categories: Nature & Environment (Nature); Locations & Architecture (Geography); Culture, Arts & Crafts (Culture); Objects & Products (Objects); Vehicles, Craft & Transportation (Transportation); Entertainment, Media & Gaming (Entertainment); Brands, Logos & Graphic Design (Brands); Sports, Gear & Venues (Sports); Notable People & Public Figures (People). The visual entities curated to evaluate atomic world knowledge range from globally recognized "head-class" landmarks and logos to specific "long-tail" biological species and artisanal artifacts. To maintain atomic isolation, each image serves as an unambiguous visual stimulus for entity naming, strictly decoupled from complex reasoning or OCR dependencies.
  • Figure 3: Category-wise F-score comparison on WorldVQA. This radar chart illustrates the performance profiles of frontier close-source and open-source MLLMs across the 8 semantic categories. The visualization highlights the relative proficiency in high-frequency domains like Sports and Brands, while revealing significant performance troughs in specialized domains such as Nature and Culture.
  • Figure 4: Entity Difficulty Distribution vs. MetaCLIP Frequency Rank Percentile across Categories. These plots illustrate the relationship between real-world entity frequency (proxied by MetaCLIP vocabulary rank percentile) and their assigned difficulty in WorldVQA. The x-axis represents the percentile rank of the entity's frequency in the MetaCLIP vocabulary, where values closer to 0 indicate high-frequency (common) entities, and higher values indicate lower-frequency (rare) entities. The left y-axis corresponds to the grey line, showing the underlying exponential density distribution of MetaCLIP word frequencies, highlighting the long-tail nature of real-world knowledge. The right y-axis shows the probability density for the fitted normal distributions of the four difficulty tiers: Trivial, Easy, Medium, and Hard.
  • Figure 5: Calibration and Confidence Distribution Analysis.Left: Reliability diagrams plotting Actual Accuracy against Stated Confidence. To ensure statistical significance, only bins containing more than 20 samples are visualized. The size of each data point is proportional to the number of samples in that bin. The black dashed diagonal (y=x) represents perfect calibration, while colored dashed lines indicate the weighted average slope for each model. Right: The distribution of stated confidence scores across the full dataset (without sample thresholding). The plots reveal a severe overconfidence trend, with most models concentrating their predictions in the 90-100% confidence range.