More Images, More Problems? A Controlled Analysis of VLM Failure Modes

TL;DR

This work systematically analyzes how current LVLMs handle multi-image reasoning and uncovers pervasive single-image biases, weak cross-image aggregation, and vulnerability to distractors. It introduces MIMIC, a controllable MS-COCO–derived benchmark to diagnose cross-image information flow and multi-concept tracking, and proposes two complementary finetuning strategies: data-centric synthetic multi-image data generation and layer-wise attention-masking with LoRA. These methods yield substantial gains across MIMIC and existing multi-image benchmarks, establishing new state-of-the-art results and demonstrating improved cross-image reasoning with significant efficiency gains. By providing a rigorous diagnostic framework, targeted remedies, and released data/code, the paper offers a practical path forward for advancing multi-image understanding in LVLMs.

Abstract

Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities, yet their proficiency in understanding and reasoning over multiple images remains largely unexplored. While existing benchmarks have initiated the evaluation of multi-image models, a comprehensive analysis of their core weaknesses and their causes is still lacking. In this work, we introduce MIMIC (Multi-Image Model Insights and Challenges), a new benchmark designed to rigorously evaluate the multi-image capabilities of LVLMs. Using MIMIC, we conduct a series of diagnostic experiments that reveal pervasive issues: LVLMs often fail to aggregate information across images and struggle to track or attend to multiple concepts simultaneously. To address these failures, we propose two novel complementary remedies. On the data side, we present a procedural data-generation strategy that composes single-image annotations into rich, targeted multi-image training examples. On the optimization side, we analyze layer-wise attention patterns and derive an attention-masking scheme tailored for multi-image inputs. Experiments substantially improved cross-image aggregation, while also enhancing performance on existing multi-image benchmarks, outperforming prior state of the art across tasks. Data and code will be made available at https://github.com/anurag-198/MIMIC.

Figures (16)

• Figure 1: Counting performance under different settings.Left (Unbalanced): We compare different LVLMs by analyzing the trade-off between the number of query images and the total number of images without controlling for number of instances. Mid (Balanced): We fix the total number of images to 7 and of object instances distributed across query images to 4,3 and 2. In both settings, performance consistently drops when instances are spread over multiple images. Right (Multi Concept): We increase the complexity by adding more classes (concepts) to the counting task, and observe a steep performance drop, indicating limited capacity for multi-concept tracking.
• Figure 2: MIMIC Bench: examples of each task.
• Figure 3: Effect of vision token sequence length on performance.Left: Sequence length reduction via 1-D pooling. The square denotes the original sequence length. Right: Control experiment reducing the information via pixel space pooling while keeping the sequence length fixed. Results are reported for counting task with 3 query images and total 10 images.
• Figure 4: Inter-image and intra-image token attention across layers. The attention patterns transitions from cross-image to intra-image interactions as we advance in depth.
• Figure 5: Answer-to-Image Attention: The baseline LLaVA OV (top row) fails to attend to the potted plant in the third image, whereas our method (bottom row) correctly focuses on the relevant object. Visualization is shown at the 15th layer of the LLM.