MMLongCite: A Benchmark for Evaluating Fidelity of Long-Context Vision-Language Models

TL;DR

MMLongCite introduces a comprehensive benchmark to evaluate the fidelity of multimodal LVLMs under long-context scenarios, addressing the gap where extended context does not ensure faithful grounding. The authors formalize problem definitions, construct tasks across image-only, interleaved image-text, and video-only contexts, and add an MMLongCite-Grounding extension to stress spatial attribution. Evaluating 12 LVLMs reveals a persistent decoupling between answer correctness and citation grounding, with grounding performance severely deteriorating as context length and visual density increase; reasoning can improve precision at the cost of recall. The work highlights critical challenges in long-context multimodal factuality and motivates future advances in grounding, retrieval, and attribution for LVLMs in real-world, long-context use cases.

Abstract

The rapid advancement of large vision language models (LVLMs) has led to a significant expansion of their context windows. However, an extended context window does not guarantee the effective utilization of the context, posing a critical challenge for real-world applications. Current evaluations of such long-context faithfulness are predominantly focused on the text-only domain, while multimodal assessments remain limited to short contexts. To bridge this gap, we introduce MMLongCite, a comprehensive benchmark designed to evaluate the fidelity of LVLMs in long-context scenarios. MMLongCite comprises 8 distinct tasks spanning 6 context length intervals and incorporates diverse modalities, including text, images, and videos. Our evaluation of state-of-the-art LVLMs reveals their limited faithfulness in handling long multimodal contexts. Furthermore, we provide an in-depth analysis of how context length and the position of crucial content affect the faithfulness of these models.

Figures (5)

• Figure 1: An overview of our proposed MMLongCite benchmark. (Left) The primary task evaluates the processing of multimodal long contexts in Image-Only, Image-Text, and Video-Only formats. (Right) MMLongCite-Grounding assesses visual grounding capabilities with scalable difficulty. In the Easy setting, models are fed a sequence of composite images, where each composite is stitched from 4 source images. In the Hard setting, models need to process a single, large composite image stitched from all source images. This often separates related evidence far apart in the patch sequence, demanding the model to resolve challenging long-range dependencies.
• Figure 2: Model Performance across different context length intervals.
• Figure 3: Model Performance on citation quality and correctness across different depths and context lengths in Visual Haystack.
• Figure 4: Prompt for evaluating citation quality with GPT-4.1.
• Figure 5: Prompt for evaluating correctness with GPT-4.1.