VL-RouterBench: A Benchmark for Vision-Language Model Routing

TL;DR

VL-RouterBench provides a comprehensive, log-driven benchmark for vision–language model routing, filling a gap in standardization for multimodal routing research. The framework constructs quality and cost matrices from inference logs, introduces an accuracy–cost trade-off via soft labels, and evaluates both feature-level and end-to-end routers across a large, diverse dataset collection. Key findings show meaningful routability gains from learned routers, with simple multimodal fusion sufficing in many cases, yet a persistent gap to an Oracle bound that invites architectural improvements in visual cues and textual structure modeling. The work offers an open-source, scalable pipeline for reproducible evaluation and drives progress toward practical, cost-aware routing in multimodal systems.

Abstract

Multi-model routing has evolved from an engineering technique into essential infrastructure, yet existing work lacks a systematic, reproducible benchmark for evaluating vision-language models (VLMs). We present VL-RouterBench to assess the overall capability of VLM routing systems systematically. The benchmark is grounded in raw inference and scoring logs from VLMs and constructs quality and cost matrices over sample-model pairs. In scale, VL-RouterBench covers 14 datasets across 3 task groups, totaling 30,540 samples, and includes 15 open-source models and 2 API models, yielding 519,180 sample-model pairs and a total input-output token volume of 34,494,977. The evaluation protocol jointly measures average accuracy, average cost, and throughput, and builds a ranking score from the harmonic mean of normalized cost and accuracy to enable comparison across router configurations and cost budgets. On this benchmark, we evaluate 10 routing methods and baselines and observe a significant routability gain, while the best current routers still show a clear gap to the ideal Oracle, indicating considerable room for improvement in router architecture through finer visual cues and modeling of textual structure. We will open-source the complete data construction and evaluation toolchain to promote comparability, reproducibility, and practical deployment in multimodal routing research.

Figures (7)

• Figure 1: Overall performance comparison on VL-RouterBench. The x-axis and y-axis are Average Cost (Avg. Cost, $\$/10$K samples) and Average Accuracy (Avg. Acc., %), respectively. Gray dots denote the performance of single models at different costs, and "Strongest" and "Cheapest" mark the baselines that use only the strongest or the cheapest model. The gray dashed curve depicts the Pareto frontier fitted from these single-model points (only models near the frontier are shown for clarity). "1st RouterDC", "2nd VLC", and "3rd MLP" indicate the top three routers by Rank Score. Points closer to the upper left reflect a better accuracy–cost trade-off. The results show that even advanced routers still have a noticeable performance gap to the Oracle in the VLM routing setting.
• Figure 2: We propose VL-RouterBench to systematically assess the overall performance of vision-language model routing strategies. The three diagrams on the left, middle, and right represent Router Data Preparation (Sec. \ref{['subsec:router_data_preparation']}), Router Training (Sec. \ref{['subsec:router_training']}), and Router Evaluation (Sec. \ref{['subsec:router_evaluation_metrics']}), respectively.
• Figure 2: Performance of Router Methods on VL-RouterBench. The best and second-best results except for Oracle are highlighted in bold and underlined, respectively. Avg. Acc. is average accuracy (%), Avg. Cost is average cost ($/10K samples), Rank is sorted by Rank Score, and Throughput is measured in $\#$K tokens/s.
• Figure 3: Dataset distribution in VL-RouterBench. The inner ring shows the three groups, and the outer ring lists the individual datasets. The numbers in parentheses indicate the number of samples contained in each entry.
• Figure 4: Performance in accuracy–cost plane. Gray dots represent baselines of individual models. Colored markers represent operating points of the same router under different $\lambda$, and curves are Pareto fronts fitted to these points. The closer to the upper left, the better the accuracy-cost trade-off.

Theorems & Definitions (1)

• proof