CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models

TL;DR

CODIS introduces a context-dependent image disambiguation benchmark for multimodal LLMs, pairing each image-question instance with two distinct free-form textual contexts to force context usage. The study evaluates 14 MLLMs and humans, revealing substantial gaps in context-aware visual understanding, with API-based models generally outperforming open-source ones and a notable Acc_p versus Acc_q disparity. Analyses highlight weaknesses in visual information extraction, pervasive biases in outputs, and strong alignment between GPT-4 and human evaluators. The work provides a rigorous benchmark and analysis framework to drive progress in context-aware visual comprehension for MLLMs, along with insights into bias mitigation and evaluator reliability.

Abstract

Multimodal large language models (MLLMs) have demonstrated promising results in a variety of tasks that combine vision and language. As these models become more integral to research and applications, conducting comprehensive evaluations of their capabilities has grown increasingly important. However, most existing benchmarks fail to consider that, in certain situations, images need to be interpreted within a broader context. In this work, we introduce a new benchmark, named as CODIS, designed to assess the ability of models to use context provided in free-form text to enhance visual comprehension. Our findings indicate that MLLMs consistently fall short of human performance on this benchmark. Further analysis confirms that these models struggle to effectively extract and utilize contextual information to improve their understanding of images. This underscores the pressing need to enhance the ability of MLLMs to comprehend visuals in a context-dependent manner. View our project website at https://thunlp-mt.github.io/CODIS.

Figures (10)

• Figure 1: Interpretation of images can be significantly influenced by contextual information. In this instance, the determination of whether the photographer was ascending or descending a staircase remains ambiguous without supplementary context (a). However, when additional information is provided, indicating the position of the greenery relative to the observer, the direction of movement of the observer becomes clear (b). In the responses, words originating from the image and the two pieces of context are highlighted in purple, blue, and green, respectively.
• Figure 2: Taxonomy of our benchmark. We show two cases for each category. In each case, we show an image and a question, along with two pieces of different context and their corresponding answers. We use Q to denote questions, C for contexts and A for answers.
• Figure 3: Distribution of five categories (left) and scenarios (right) of our CODIS benchmark.
• Figure 4: Three cases to show alterations in model outputs resulting from the removal of context information.
• Figure 5: Three cases to show alterations in model outputs when replacing images with captions.