AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs

TL;DR

AMVICC introduces a cross-modal benchmark to profile failure modes in both vision-language models and image generation models by aligning prompts and tasks with the MMVP benchmark. It systematically analyzes 11 MLLMs and 3 IGMs across nine visual-reasoning categories, revealing both shared and modality-specific failures and offering insights into how architecture and prompting strategies influence elementary visual understanding. The study demonstrates that explicit prompts challenge IGMs more than implicit prompts and highlights cross-modal alignment as a crucial direction for improving unified vision-language systems. By providing a framework for cross-modal evaluation and ablations on prompting and stochasticity, AMVICC lays groundwork for targeted improvements in both interpretation and generation tasks with practical implications for evaluation and model development.

Abstract

We investigated visual reasoning limitations of both multimodal large language models (MLLMs) and image generation models (IGMs) by creating a novel benchmark to systematically compare failure modes across image-to-text and text-to-image tasks, enabling cross-modal evaluation of visual understanding. Despite rapid growth in machine learning, vision language models (VLMs) still fail to understand or generate basic visual concepts such as object orientation, quantity, or spatial relationships, which highlighted gaps in elementary visual reasoning. By adapting MMVP benchmark questions into explicit and implicit prompts, we create \textit{AMVICC}, a novel benchmark for profiling failure modes across various modalities. After testing 11 MLLMs and 3 IGMs in nine categories of visual reasoning, our results show that failure modes are often shared between models and modalities, but certain failures are model-specific and modality-specific, and this can potentially be attributed to various factors. IGMs consistently struggled to manipulate specific visual components in response to prompts, especially in explicit prompts, suggesting poor control over fine-grained visual attributes. Our findings apply most directly to the evaluation of existing state-of-the-art models on structured visual reasoning tasks. This work lays the foundation for future cross-modal alignment studies, offering a framework to probe whether generation and interpretation failures stem from shared limitations to guide future improvements in unified vision-language modeling.

Figures (4)

• Figure 1: Comparison of implicit prompts to explicit prompts for the 2 pictures in a pair
• Figure 2: Diagram of the AMVICC Creation Pipeline: We created implicit prompts based off of the general scenario introduced by the question and created explicit prompts by adding specifics in line with the specific answer choice for that image ID.
• Figure 3: Benchmark results of current MLLMs: We evaluate pair accuracy across 11 models based on the questions and images from the MMVP dataset.
• Figure 4: Examples of specific IGMs' abilities to generate an image based on explicit prompts. We handpick 5 out of the 300 questions in the MMVP dataset to delineate disparities between the models. It is apparent that Google: Gemini 2.5 Flash Image was the most accurate, followed by OpenAI: DALL·E 3, and Stability AI: Stable Diffusion 3.5 Large, in that order. An important thing to note is that the IGMs don't directly state Yes or No or any of the answer choices, for that matter. However, based on the models' image generation, we can associate certain answer choices with the models. A ✓ indicates that the model generated an image in accordance with the given prompt, whereas an × indicates the opposite.