AdaptMMBench: Benchmarking Adaptive Multimodal Reasoning for Mode Selection and Reasoning Process

TL;DR

AdaptMMBench introduces a domain-diverse benchmark to assess adaptive multimodal reasoning in vision-language tasks, explicitly separating the meta-cognitive mode selection from the downstream reasoning process. It defines data samples, annotations, and five domains (real-world, OCR, GUI, knowledge, math) with both text-only and adaptive-solution instances, and uses MCC to measure mode-selection accuracy while evaluating reasoning process via key-step coverage, tool execution fidelity, and efficiency. Findings show mode selection improves with model capacity but correlates weakly with final accuracy, whereas reasoning process quality aligns more with performance, and tool effectiveness varies significantly across architectures. The benchmark provides a framework for diagnosing adaptive reasoning bottlenecks and guiding future improvements in both tool invocation strategies and reasoning reliability, with potential impact on designing efficient, dynamic VLM systems.

Abstract

Adaptive multimodal reasoning has emerged as a promising frontier in Vision-Language Models (VLMs), aiming to dynamically modulate between tool-augmented visual reasoning and text reasoning to enhance both effectiveness and efficiency. However, existing evaluations rely on static difficulty labels and simplistic metrics, which fail to capture the dynamic nature of difficulty relative to varying model capacities. Consequently, they obscure the distinction between adaptive mode selection and general performance while neglecting fine-grained process analyses. In this paper, we propose AdaptMMBench, a comprehensive benchmark for adaptive multimodal reasoning across five domains: real-world, OCR, GUI, knowledge, and math, encompassing both direct perception and complex reasoning tasks. AdaptMMBench utilizes a Matthews Correlation Coefficient (MCC) metric to evaluate the selection rationality of different reasoning modes, isolating this meta-cognition ability by dynamically identifying task difficulties based on models' capability boundaries. Moreover, AdaptMMBench facilitates multi-dimensional process evaluation across key step coverage, tool effectiveness, and computational efficiency. Our evaluation reveals that while adaptive mode selection scales with model capacity, it notably decouples from final accuracy. Conversely, key step coverage aligns with performance, though tool effectiveness remains highly inconsistent across model architectures.

Figures (17)

• Figure 1: Comparative Analysis of Accuracy, Reasoning Mode Selection, and Reasoning Process. Closed-source models achieve stronger performance in accuracy and mode selection, while reasoning process quality is analyzed on open-source models due to limited access to closed-source reasoning traces.
• Figure 2: Illustration of our model-specific difficulty evaluation. Existing methods rely on static difficulty levels, while difficulty is inherently model-dependent.
• Figure 3: An Overview of AdaptMMBench. The benchmark contains data from five domains. Each domain includes samples requiring zoom-in and enhancement tools. We annotate zoom-in regions, enhancement arguments, and key reasoning steps.
• Figure 4: Domains and category of AdaptMMBench.
• Figure 5: Evaluation pipeline for mode selection and reasoning process.