ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks

TL;DR

ProBench introduces an automatic, open-ended benchmark for evaluating multimodal foundation models on expert tasks. It aggregates 4,000 queries across 10 fields and 56 sub-fields, supports 17 languages and conversations up to 13 turns, and benchmarks 24 MLLMs using an MLLM-as-a-Judge with an Elo-style leaderboard augmented by a Bradley-Terry model for style bias control ($α=400$, $K=32$). The results show that strong open-source models can rival proprietary ones, yet the benchmark reveals persistent challenges in visual perception, domain knowledge, and long-context reasoning, especially under multilingual and multi-round settings; human agreement with the judge reaches 79.9%. A distilled local evaluator based on Llama-3.2-11B-Vision-Instruct is proposed to enable cost-efficient, private evaluation. Overall, ProBench provides a robust framework to align MLLM development with high-value professional tasks and real-world decision-making.

Abstract

Solving expert-level multimodal tasks is a key milestone towards general intelligence. As the capabilities of multimodal large language models (MLLMs) continue to improve, evaluation of such advanced multimodal intelligence becomes necessary yet challenging. In this work, we introduce ProBench, a benchmark of open-ended user queries that require professional expertise and advanced reasoning. ProBench consists of 4,000 high-quality samples independently submitted by professionals based on their daily productivity demands. It spans across 10 fields and 56 sub-fields, including science, arts, humanities, coding, mathematics, and creative writing. Experimentally, we evaluate and compare 24 latest models using MLLM-as-a-Judge. Our results reveal that although the best open-source models rival the proprietary ones, ProBench presents significant challenges in visual perception, textual understanding, domain knowledge and advanced reasoning, thus providing valuable directions for future multimodal AI research efforts.

Figures (11)

• Figure 1: Examples of ProBench. Our ProBench spans 10 task fields and 56 sub-fields, supports 17 languages, and supports conversations with up to 13 conversation turns. We show the task and image fields in the header of each sample. We use 'Engineering Drawings' for 'Engineering and Technical Drawings' in the first plot of the second row due to space constraints. More diverse and longer samples are provided in the supplementary material.
• Figure 2: Comparison with WildVision lu2024wildvision on challenge levels of (a) text, (b) image, and (c) reasoning for user instruction queries. To ensure a fair comparison, we follow WildVision by selecting the top 500 highest-quality queries from the single-round conversations. It can be seen that ProBench contains significantly more hard samples than WildVision.
• Figure 3: ProBench overview. Distributions of (a) task fields on the single-round track, (b) languages on the multi-linguistic track, and (c) conversation rounds on the multi-round tracks.
• Figure 4: Framework of ProBench. Starting with 100K crowdsourced conversations, we identify high-quality user queries to curate single-round, multi-linguistic, and multi-round tracks. Using MLLM-as-a-Judge, we benchmark and rank 24 state-of-the-art MLLMs with ELO ratings. To ensure fairness, the ELO ratings are de-biased to remove confounder effects (e.g., MLLM response formats), resulting in the final ProBench leaderboard. Icons in the figure are sourced from flaticon2025.
• Figure 5: Example queries from ProBench. As shown, significant domain knowledge and reasoning capabilities are needed to solve ProBench queries. For brevity, we only show examples with relatively shorter text queries, with the remark that longer queries are common in ProBench. More examples can be found in the appendix.