MMRA: A Benchmark for Evaluating Multi-Granularity and Multi-Image Relational Association Capabilities in Large Visual Language Models

TL;DR

MMRA introduces a ConceptNet-based benchmark to evaluate how LVLMs reason about relations across two images at entity and image granularities. It combines 11 subtasks across two levels, curated from 1,024 annotated image pairs, with careful leakage mitigation and four input configurations to probe perception and reasoning. The results reveal strong gaps in fine-grained and spatial multi-image understanding, limited gains from image descriptions, and a bottleneck in the language-model reasoning component, especially for sequence-related tasks. These findings underscore the need for improved fine-grained perception and cross-image sequence modeling to advance multi-image LVLM capabilities and robust cross-modal reasoning in practical settings.

Abstract

Given the remarkable success that large visual language models (LVLMs) have achieved in image perception tasks, the endeavor to make LVLMs perceive the world like humans is drawing increasing attention. Current multi-modal benchmarks primarily focus on facts or specific topic-related knowledge contained within individual images. However, they often overlook the associative relations between multiple images, which require the identification and analysis of similarities among entities or content present in different images. Therefore, we propose the multi-image relation association task and a meticulously curated Multi-granularity Multi-image Relational Association (MMRA) benchmark, comprising 1,024 samples. In order to systematically and comprehensively evaluate current LVLMs, we establish an associational relation system among images that contain 11 subtasks (e.g, UsageSimilarity, SubEvent) at two granularity levels (i.e., image and entity) according to the relations in ConceptNet. Our experiments reveal that on the MMRA benchmark, current multi-image LVLMs exhibit distinct advantages and disadvantages across various subtasks. Notably, fine-grained, entity-level multi-image perception tasks pose a greater challenge for LVLMs compared to image-level tasks. Moreover, LVLMs perform poorly on spatial-related tasks, indicating that LVLMs still have limited spatial awareness. Additionally, our findings indicate that while LVLMs demonstrate a strong capability to perceive image details, enhancing their ability to associate information across multiple images hinges on improving the reasoning capabilities of their language model component. Moreover, we explored the ability of LVLMs to perceive image sequences within the context of our multi-image association task. Our experiments show that the majority of current LVLMs do not adequately model image sequences during the pre-training process.

Figures (7)

• Figure 1: Overview of the MMRA benchmark. Left: image Associational Relations extended from the ConceptNet; Right: the examples of Multi-Image Relation Association task.
• Figure 2: The process of annotation.
• Figure 3: The number and ratio of each subtask in our MMRA benchmark. The integers in the graph represent the number of samples in each task, while the percentages in parentheses indicate the proportion of each task in the entire benchmark.
• Figure 4: Comparing results before and after textual answer leakage elimination.
• Figure 5: The relative improvement of LVLMs on MMRA benchmark.