MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly

TL;DR

MMLongBench provides a comprehensive, length-controlled benchmark to evaluate long-context vision-language models across five downstream task categories with 13,331 examples and five standardized input lengths (8K–128K tokens). It unifies cross-modal token counting and covers diverse image types, enabling thorough comparisons of 46 models (open- and closed-source). Key findings show that single-task performance poorly predicts overall long-context ability, OCR and cross-modal retrieval are major bottlenecks, and models with stronger reasoning consistently fare better in long-context settings. The results emphasize the need for broad, multi-task evaluation to guide future improvements in token encoding, position extrapolation, and multi-modal reasoning. MMLongBench thus provides a foundational, extensible platform for diagnosing and accelerating progress in LCVLM development.

Abstract

The rapid extension of context windows in large vision-language models has given rise to long-context vision-language models (LCVLMs), which are capable of handling hundreds of images with interleaved text tokens in a single forward pass. In this work, we introduce MMLongBench, the first benchmark covering a diverse set of long-context vision-language tasks, to evaluate LCVLMs effectively and thoroughly. MMLongBench is composed of 13,331 examples spanning five different categories of downstream tasks, such as Visual RAG and Many-Shot ICL. It also provides broad coverage of image types, including various natural and synthetic images. To assess the robustness of the models to different input lengths, all examples are delivered at five standardized input lengths (8K-128K tokens) via a cross-modal tokenization scheme that combines vision patches and text tokens. Through a thorough benchmarking of 46 closed-source and open-source LCVLMs, we provide a comprehensive analysis of the current models' vision-language long-context ability. Our results show that: i) performance on a single task is a weak proxy for overall long-context capability; ii) both closed-source and open-source models face challenges in long-context vision-language tasks, indicating substantial room for future improvement; iii) models with stronger reasoning ability tend to exhibit better long-context performance. By offering wide task coverage, various image types, and rigorous length control, MMLongBench provides the missing foundation for diagnosing and advancing the next generation of LCVLMs.

Figures (33)

• Figure 1: Performance on MMLongBench. We report results for selected frontier models, and the full results of all models are provided in \ref{['fig:results_length_full']}. Note that Claude-3.7-Sonnet supports at most 100 images, and we mark the results as N/A for cases with more images (More in \ref{['app:claude']})
• Figure 2: Distribution of long-document VQA (DocVQA) with respect to performance on MM-NIAH variants. We find that the models are concentrated in the coral-shaded areas.
• Figure 3: Model performance on VH-Multi dataset. Random guess yields 50% accuracy, highlighting its difficulty.
• Figure 4: Spearman's $\rho$ across all 46 models at 128K tokens.
• Figure 5: Spearman's $\rho$ between all categories with $L=$128K. For each category, the Avg excludes the correlation with itself.