VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning

TL;DR

VOILA tackles the gap in evaluating high-level perceptual and relational reasoning in multimodal models with an open-ended, dynamic visual analogy benchmark. It combines a generation-based data-creation pipeline using SDXL to produce millions of questions across structured rule configurations, yielding VOILA-WD and VOILA-ND variants to modulate difficulty. Empirical results reveal a large human-machine gap and show that techniques like least-to-most prompting and sequential image input improve reasoning but fall far short of human levels, especially in applying abstract relations. By providing a rigorous framework for diagnosing and improving abstract visual reasoning in multimodal systems, VOILA guides future work toward stronger relational transfer and reasoning capabilities.

Abstract

Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information. Despite their exceptional performance on visual understanding benchmarks, measuring their ability to reason abstractly across multiple images remains a significant challenge. To address this, we introduce VOILA, a large-scale, open-ended, dynamic benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning. VOILA employs an analogical mapping approach in the visual domain, requiring models to generate an image that completes an analogy between two given image pairs, reference and application, without relying on predefined choices. Our experiments demonstrate that the analogical reasoning tasks in VOILA present a challenge to MLLMs. Through multi-step analysis, we reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning. Notably, we observe that performance improves when following a multi-step strategy of least-to-most prompting. Comprehensive evaluations on open-source models and GPT-4o show that on text-based answers, the best accuracy for challenging scenarios is 13% (LLaMa 3.2) and even for simpler tasks is only 29% (GPT-4o), while human performance is significantly higher at 70% across both difficulty levels.

Figures (24)

• Figure 1: Examples of visual analogy questions from the VOILA benchmark with distractions (VOILA-WD) and without distractions (VOILA-ND). The aim is to generate an image that completes the analogy problem following perceptual and relational reasoning. Each visual analogy question has a specific rule configuration that leads to the answer. The questions in VOILA-WD benchmark can apply the distraction rule when no relational pattern exists between images. For the examples shown above, the answers that complete the VOILA-ND analogy by following the relation rules are "two bears driving a car" and "two female children swimming". The answers to the VOILA-WD analogy are "four [any subject] reading" and "two male children doing anything".
• Figure 2: Dataset creation pipeline of VOILA.
• Figure 3: VOILA multi-step reasoning and evaluation pipeline. The top section illustrates two visual input formats. The left side of the MLLMs connection displays the four primary tasks along with their corresponding prompts, while the right side presents the expected outcomes for each task. The results are scored in the evaluation stage utilizing GPT-4o and ground truths.
• Figure 4: Accuracy of VOILA-WD and VOILA-ND at each step, respectively.
• Figure 5: Accuracy of baseline MLLMs regarding properties at each step on VOILA-WD.