R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation

TL;DR

R-Bench is a graduate-level, multilingual, multidisciplinary benchmark for evaluating complex reasoning in LLMs and MLLMs. It collects 1,094 language questions across 108 subjects and 665 multimodal questions across 83 subjects from 19 departments, with a rigorous data pipeline, expert and model screening, and bilingual translations to ensure cross-language difficulty and rigor. Experimental results show current models struggle with multimodal reasoning and cross-disciplinary problems, with Chain-of-Thought prompting helping many models but not all, and clear saturation in some language-model combinations. The work provides publicly available data and code to enable robust, cross-disciplinary reasoning evaluation and to guide future model development toward more generalizable, multimodal reasoning capabilities. It emphasizes that true progression requires broad, multilingual benchmarks that capture diverse domains and modalities beyond traditional single-domain tests.

Abstract

Reasoning stands as a cornerstone of intelligence, enabling the synthesis of existing knowledge to solve complex problems. Despite remarkable progress, existing reasoning benchmarks often fail to rigorously evaluate the nuanced reasoning capabilities required for complex, real-world problemsolving, particularly in multi-disciplinary and multimodal contexts. In this paper, we introduce a graduate-level, multi-disciplinary, EnglishChinese benchmark, dubbed as Reasoning Bench (R-Bench), for assessing the reasoning capability of both language and multimodal models. RBench spans 1,094 questions across 108 subjects for language model evaluation and 665 questions across 83 subjects for multimodal model testing in both English and Chinese. These questions are meticulously curated to ensure rigorous difficulty calibration, subject balance, and crosslinguistic alignment, enabling the assessment to be an Olympiad-level multi-disciplinary benchmark. We evaluate widely used models, including OpenAI o1, GPT-4o, DeepSeek-R1, etc. Experimental results indicate that advanced models perform poorly on complex reasoning, especially multimodal reasoning. Even the top-performing model OpenAI o1 achieves only 53.2% accuracy on our multimodal evaluation. Data and code are made publicly available at here.

Figures (6)

• Figure 1: Top-1 accuracy comparison of different models on MMLU, MMMU, and $\mathbf{\mathcal{R}}$-Bench. $\mathbf{\mathcal{R}}$-Bench poses a greater challenge to current models.
• Figure 2: Pipeline of building $\mathbf{\mathcal{R}}$-Bench. The process is divided into six steps, which are detailed in Sec. \ref{['method']}. The funnel represents screening. We always filter out the blue ball and preserve the brown one. In Step 2, KQ and RQ denote knowledge-based questions and reasoning-based questions, respectively. In Step 4, $\textless$ 2000 indicates that the reasoning tokens of o1 are less than 2000. Finally, in Step 5, AQ and CQ represent ambiguous questions and clear questions, respectively. -T indicates text-only testing for LLMs. -M means multimodal testing. zh represents the Chinese version.
• Figure 3: Some examples in $\mathbf{\mathcal{R}}$-Bench. These examples show that $\mathbf{\mathcal{R}}$-Bench is multidisciplinary, multimodal, and multilingual. As shown in the figure, the problems in $\mathbf{\mathcal{R}}$-Bench are complex and cannot be solved by quick thinking, which shows that $\mathbf{\mathcal{R}}$-Bench focuses on deep reasoning problems rather than knowledge problems, such as conceptual problems.
• Figure 4: According to statistics on $\mathbf{\mathcal{R}}$-Bench, the benchmark spans 19 departments, including mathematics, physics, biology, computer science, and chemistry, covering over 100 subjects such as Inorganic Chemistry, Chemical Reaction Kinetics, and Electromagnetism. It features 1,094 questions designed for testing language models and 665 questions specifically tailored for evaluating multimodal reasoning capabilities. For a detailed list of subjects, please refer to the appendix.
• Figure 5: The performance of different models on questions of the same difficulty in Chinese and English.