MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing

TL;DR

MMR-Bench tackles the problem of routing among a heterogeneous set of multimodal LLMs under budget constraints. It introduces an offline, cost-aware benchmark with modality-aware inputs, precomputed utilities and costs, and a suite of routing policies ranging from unimodal to multimodal cues, including matrix-factorization–based approaches that generalize across tasks. Empirical results show that multimodal signals and adaptive fusion significantly improve the cost–accuracy frontier, with routed systems matching or surpassing the strongest single model at roughly a third of its cost, and policies trained on subset data generalizing to new domains and text-only tasks. The benchmark provides a standardized platform for studying adaptive multimodal model selection and practical deployment of MLLMs under realistic budgets, with strong implications for efficient, scalable AI systems.

Abstract

Multimodal large language models (MLLMs) have advanced rapidly, yet heterogeneity in architecture, alignment strategies, and efficiency means that no single model is uniformly superior across tasks. In practical deployments, workloads span lightweight OCR to complex multimodal reasoning; using one MLLM for all queries either over-provisions compute on easy instances or sacrifices accuracy on hard ones. Query-level model selection (routing) addresses this tension, but extending routing from text-only LLMs to MLLMs is nontrivial due to modality fusion, wide variation in computational cost across models, and the absence of a standardized, budget-aware evaluation. We present MMR-Bench, a unified benchmark that isolates the multimodal routing problem and enables comparison under fixed candidate sets and cost models. MMR-Bench provides (i) a controlled environment with modality-aware inputs and variable compute budgets, (ii) a broad suite of vision-language tasks covering OCR, general VQA, and multimodal math reasoning, and (iii) strong single-model reference, oracle upper bounds, and representative routing policies. Using MMR-Bench, we show that incorporating multimodal signals improves routing quality. Empirically, these cues improve the cost-accuracy frontier and enable the routed system to exceed the strongest single model's accuracy at roughly 33% of its cost. Furthermore, policies trained on a subset of models and tasks generalize zero-shot to new datasets and text-only benchmarks without retuning, establishing MMR-Bench as a foundation for studying adaptive multimodal model selection and efficient MLLM deployment. The code will be available at: https://github.com/Hunter-Wrynn/MMR-Bench.

Figures (6)

• Figure 1: The workflow of MLLM routing. It can be applied to various tasks such as mathematical reasoning, multimodal recommendation and understanding, and medical diagnosis.
• Figure 2: Routing comparison across different modality signals, where Qwen denotes the Qwen2.5-VL series and InternVL denotes the InternVL3 series.
• Figure 3: Composition of MMR-Bench across scenarios and datasets.
• Figure 4: Cost-accuracy Pareto frontiers on MMR-Bench. Lines of different styles represent different routing strategies, while single-model positions are indicated by their logos. For readability, the x-axis is log-scaled to emphasize the low-cost region. Compared with any fixed model, routing shifts the frontier upward and leftward, indicating higher accuracy at the same cost and lower cost at the same accuracy.
• Figure S1: Cost-Accuracy Pareto Frontiers across Six Datasets. The curves illustrate the trade-off between normalized cost (x-axis) and accuracy (y-axis). Routing strategies (colored lines) consistently envelop the single-model baselines (stars), demonstrating superior efficiency. The top row represents mathematical reasoning tasks, while the bottom row covers general VQA and OCR scenarios.

Theorems & Definitions (1)

• Definition 3.1: MLLM routing