Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Pixels to Facts (Pix2Fact): Benchmarking Multi-Hop Reasoning for Fine-Grained Visual Fact Checking

Yifan Jiang, Cong Zhang, Bofei Zhang, Yifan Yang, Bingzhang Wang, Yew-Soon Ong

TL;DR

Pix2Fact tackles the need for evaluating vision-language models on tasks that require both fine-grained visual grounding and knowledge-intensive, multi-hop reasoning in real-world settings. It introduces a 1,000-image, 4K+ benchmark across eight daily-life categories, with expert-crafted (Q,A) pairs that demand precise localization and retrieval from external knowledge. Empirical results across nine SOTA VLMs reveal a substantial gap to human performance (top ~27% vs ~56%), with visual grounding identified as the primary bottleneck and open-world knowledge access providing the strongest gains. The benchmark aims to drive the development of next-generation multimodal agents capable of robust perception and reasoning in unstructured environments.

Abstract

Despite progress on general tasks, VLMs struggle with challenges demanding both detailed visual grounding and deliberate knowledge-based reasoning, a synergy not captured by existing benchmarks that evaluate these skills separately. To close this gap, we introduce Pix2Fact, a new visual question-answering benchmark designed to evaluate expert-level perception and knowledge-intensive multi-hop reasoning. Pix2Fact contains 1,000 high-resolution (4K+) images spanning 8 daily-life scenarios and situations, with questions and answers meticulously crafted by annotators holding PhDs from top global universities working in partnership with a professional data annotation firm. Each question requires detailed visual grounding, multi-hop reasoning, and the integration of external knowledge to answer. Our evaluation of 9 state-of-the-art VLMs, including proprietary models like Gemini-3-Pro and GPT-5, reveals the substantial challenge posed by Pix2Fact: the most advanced model achieves only 24.0% average accuracy, in stark contrast to human performance of 56%. This significant gap underscores the limitations of current models in replicating human-level visual comprehension. We believe Pix2Fact will serve as a critical benchmark to drive the development of next-generation multimodal agents that combine fine-grained perception with robust, knowledge-based reasoning.

From Pixels to Facts (Pix2Fact): Benchmarking Multi-Hop Reasoning for Fine-Grained Visual Fact Checking

TL;DR

Pix2Fact tackles the need for evaluating vision-language models on tasks that require both fine-grained visual grounding and knowledge-intensive, multi-hop reasoning in real-world settings. It introduces a 1,000-image, 4K+ benchmark across eight daily-life categories, with expert-crafted (Q,A) pairs that demand precise localization and retrieval from external knowledge. Empirical results across nine SOTA VLMs reveal a substantial gap to human performance (top ~27% vs ~56%), with visual grounding identified as the primary bottleneck and open-world knowledge access providing the strongest gains. The benchmark aims to drive the development of next-generation multimodal agents capable of robust perception and reasoning in unstructured environments.

Abstract

Despite progress on general tasks, VLMs struggle with challenges demanding both detailed visual grounding and deliberate knowledge-based reasoning, a synergy not captured by existing benchmarks that evaluate these skills separately. To close this gap, we introduce Pix2Fact, a new visual question-answering benchmark designed to evaluate expert-level perception and knowledge-intensive multi-hop reasoning. Pix2Fact contains 1,000 high-resolution (4K+) images spanning 8 daily-life scenarios and situations, with questions and answers meticulously crafted by annotators holding PhDs from top global universities working in partnership with a professional data annotation firm. Each question requires detailed visual grounding, multi-hop reasoning, and the integration of external knowledge to answer. Our evaluation of 9 state-of-the-art VLMs, including proprietary models like Gemini-3-Pro and GPT-5, reveals the substantial challenge posed by Pix2Fact: the most advanced model achieves only 24.0% average accuracy, in stark contrast to human performance of 56%. This significant gap underscores the limitations of current models in replicating human-level visual comprehension. We believe Pix2Fact will serve as a critical benchmark to drive the development of next-generation multimodal agents that combine fine-grained perception with robust, knowledge-based reasoning.
Paper Structure (24 sections, 5 figures, 7 tables)

This paper contains 24 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pix2Fact
  4. An Overview of Pix2Fact
  5. Image Data Curation Process
  6. $(Q,A)$ Pair Construction Process
  7. Comparisons with Existing Benchmarks
  8. Benchmarking SOTA VLMs
  9. Model Configurations
  10. Main Results
  11. Contest with Human Experts
  12. Conclusion
  13. Acknowledgment
  14. Results for Contest with Human Experts.
  15. LLM Prompt
  16. ...and 9 more sections

Figures (5)

  • Figure 1: The scenarios and examples of Pix2Fact.
  • Figure 2: The process of constructing Pix2Fact benchmark.
  • Figure 3: Statistics of Pix2Fact.
  • Figure 4: Result analysis for Gemini-3-Pro with web-search function.
  • Figure 5: Result analysis for Gemini-3-Pro without web-search function.