Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Res-Bench: Benchmarking the Robustness of Multimodal Large Language Models to Dynamic Resolution Input

Chenxu Li, Zhicai Wang, Yuan Sheng, Xingyu Zhu, Yanbin Hao, Xiang Wang

TL;DR

This work addresses the neglected problem of resolution robustness in multimodal large language models (MLLMs) by introducing Res-Bench, a benchmark with $14{,}400$ samples across $12$ input resolutions and $6$ capability dimensions, paired with robustness metrics including $Acc_{avg}$, $\rho$, $ACE$, and $RCE$. It reveals that current state-of-the-art models exhibit substantial instability when input resolution changes, driven by architectural choices like native dynamic processing versus patch-based methods and by preprocessing strategies. The authors demonstrate that preprocessing (padding versus super-resolution) and diffusion-based SR can mitigate some degradation, and that mixed-resolution fine-tuning can substantially improve both accuracy and stability, with demonstrated gains in Instance Reasoning and reduced ACE/RCE. Overall, Res-Bench provides a valuable framework for evaluating and guiding improvements in resolution robustness for MLLMs, with practical implications for developing more reliable, adaptable multimodal intelligence.

Abstract

Multimodal Large Language Models (MLLMs) increasingly support dynamic image resolutions. However, current evaluation paradigms primarily assess semantic performance, overlooking the critical question of resolution robustness - whether performance remains stable across varying input resolutions. To address this gap, we introduce \textbf{Res-Bench}, a comprehensive benchmark comprising 14,400 samples across 12 resolution levels and six core capability dimensions. We designed a novel evaluation framework that goes beyond traditional accuracy metrics to capture performance stability. This framework introduces multiple robustness metrics: Spearman's correlation for assessing resolution-performance trends, and Absolute/Relative Continuous Error (ACE/RCE) for measuring performance volatility. Using these metrics, we conducted a large-scale evaluation of leading MLLMs. Our analysis encompasses: (1) model-centric and task-centric robustness examination, (2) investigation of preprocessing strategies including padding and super-resolution, and (3) exploration of fine-tuning for stability enhancement.

Res-Bench: Benchmarking the Robustness of Multimodal Large Language Models to Dynamic Resolution Input

TL;DR

This work addresses the neglected problem of resolution robustness in multimodal large language models (MLLMs) by introducing Res-Bench, a benchmark with samples across input resolutions and capability dimensions, paired with robustness metrics including , , , and . It reveals that current state-of-the-art models exhibit substantial instability when input resolution changes, driven by architectural choices like native dynamic processing versus patch-based methods and by preprocessing strategies. The authors demonstrate that preprocessing (padding versus super-resolution) and diffusion-based SR can mitigate some degradation, and that mixed-resolution fine-tuning can substantially improve both accuracy and stability, with demonstrated gains in Instance Reasoning and reduced ACE/RCE. Overall, Res-Bench provides a valuable framework for evaluating and guiding improvements in resolution robustness for MLLMs, with practical implications for developing more reliable, adaptable multimodal intelligence.

Abstract

Multimodal Large Language Models (MLLMs) increasingly support dynamic image resolutions. However, current evaluation paradigms primarily assess semantic performance, overlooking the critical question of resolution robustness - whether performance remains stable across varying input resolutions. To address this gap, we introduce \textbf{Res-Bench}, a comprehensive benchmark comprising 14,400 samples across 12 resolution levels and six core capability dimensions. We designed a novel evaluation framework that goes beyond traditional accuracy metrics to capture performance stability. This framework introduces multiple robustness metrics: Spearman's correlation for assessing resolution-performance trends, and Absolute/Relative Continuous Error (ACE/RCE) for measuring performance volatility. Using these metrics, we conducted a large-scale evaluation of leading MLLMs. Our analysis encompasses: (1) model-centric and task-centric robustness examination, (2) investigation of preprocessing strategies including padding and super-resolution, and (3) exploration of fine-tuning for stability enhancement.
Paper Structure (25 sections, 5 equations, 19 figures, 13 tables)

This paper contains 25 sections, 5 equations, 19 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Res-Bench
  4. Data Collection Process
  5. Comprehensive Dataset Analysis
  6. Evaluation Metrics
  7. Experiment
  8. Experiment Analysis
  9. Analysis of Preprocessing Strategies
  10. Enhancing Robustness via Fine-tuning
  11. Limitations and Future Work
  12. Conclusion
  13. Appendix
  14. Discussions
  15. Social Impacts
  16. ...and 10 more sections

Figures (19)

  • Figure 1: An Example of ResBench. There is a lack of resolution robustness in MLLMs, where models' performance on the given task fluctuates unpredictably as the input resolution increases.
  • Figure 2:
  • Figure 3: Task-level Results. We show the result with different input resolution across six core tasks. To better show the difference of resolution robustness among tasks, we show the results of linear regression on all data points in each subplot. CP: Coarse-grained Perception, FP: Fine-grained Perception, IR: Instance Reasoning, LR: Logical Reasoning, Math: Mathematical Reasoning, OCR: Optical Character Recognition.
  • Figure 4: Example of Preprocessing. (a) Image padding by adding non-imformation pixels around images. (b) Image super-resolution using off-the-shelf SR models.
  • Figure 5: An example of the "Image Style" sub-task from the "Coarse-grained Perception" capability dimension.
  • ...and 14 more figures