JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images

TL;DR

JourneyBench presents a large, human-annotated vision-language benchmark built on generated imagery to rigorously assess fine-grained multimodal reasoning across five tasks, including MCOT, MMCOT, imaginary captioning, HaloQuest, and fine-grained retrieval. The authors implement a novel HMIL data-generation and adversarial filtering pipeline to produce challenging, sample-specific distractors and complementary visual cues that force models to rely on true multimodal understanding rather than language priors. Across five tasks, current state-of-the-art models exhibit substantial gaps, especially in cross-modal co-reference, external-knowledge reasoning, and hallucination control, underscoring the need for more robust multimodal reasoning. By releasing 13.6K image-text samples and detailed evaluation protocols, JourneyBench provides a comprehensive, hard-to-game benchmark with practical implications for deploying reliable multimodal AI systems in real-world settings.

Abstract

Existing vision-language understanding benchmarks largely consist of images of objects in their usual contexts. As a consequence, recent multimodal large language models can perform well with only a shallow visual understanding by relying on background language biases. Thus, strong performance on these benchmarks does not necessarily correlate with strong visual understanding. In this paper, we release JourneyBench, a comprehensive human-annotated benchmark of generated images designed to assess the model's fine-grained multimodal reasoning abilities across five tasks: complementary multimodal chain of thought, multi-image VQA, imaginary image captioning, VQA with hallucination triggers, and fine-grained retrieval with sample-specific distractors. Unlike existing benchmarks, JourneyBench explicitly requires fine-grained multimodal reasoning in unusual imaginary scenarios where language bias and holistic image gist are insufficient. We benchmark state-of-the-art models on JourneyBench and analyze performance along a number of fine-grained dimensions. Results across all five tasks show that JourneyBench is exceptionally challenging for even the best models, indicating that models' visual reasoning abilities are not as strong as they first appear. We discuss the implications of our findings and propose avenues for further research.

Figures (22)

• Figure 1: JourneyBench Tasks with Fine-grained Categories and Example Data. JourneyBench includes five fundamental vision-language understanding tasks with unconventional imaginary images to test the limits of models' biases, hallucination tendencies, and fine-grained perception abilities.
• Figure 2: Examples of Imaginary Images and Human-Machine-in-the-Loop Pipeline.
• Figure 3: Comparison between ScienceQA, MathVista (left), and our Strictly Complementary MCOT (right) with Examples. While ScienceQA and MathVista images provide redundant visual information, Journeybench provides complimentary visual information that is necessary to answer the question. This ensures a more rigorous evaluation of multimodal reasoning capabilities.
• Figure 4: Zero-shot Evaluation on Fine-grained Categories of Retrieval and Captioning. I2T (left), T2I (center), and Imaginary Image Captioning (right) are measured by Recall@1, Recall@1, and CIDEr respectively. Models particularly struggle with "Unusual Construction" subcategory.
• Figure 5: Zero-shot Evaluation on Fine-grained Categories of MCOT. Models struggle to get high accuracy in all categories, especially for image-question pairs with hallucinations or with large numbers of objects.