STEMVerse: A Dual-Axis Diagnostic Framework for STEM Reasoning in Large Language Models

TL;DR

STEMVerse shifts STEM evaluation from single-score benchmarks to a dual-axis capability map that jointly encodes academic specialization and cognitive complexity. By re-aggregating $20{,}374$ problems across Mathematics, Physics, Chemistry, and Biology into a Discipline × Cognition space with $27$ sub-disciplines and $6$ Bloom’s levels, it enables fine-grained diagnostics of where LLMs fail—knowledge gaps versus high-order reasoning gaps. Across open-source model families from $3$B to $14$B, STEMVerse reveals non-linear scaling, a logic-symbolic collapse in symbolic fields, and an instruction-tuning paradox that can degrade high-order reasoning. The framework offers a principled, actionable roadmap for diagnosing and ultimately improving STEM reasoning in LLMs, guiding targeted training and evaluation strategies.

Abstract

As Large Language Models (LLMs) achieve significant breakthroughs in complex reasoning tasks, evaluating their proficiency in science, technology, engineering, and mathematics (STEM) has become a primary method for measuring machine intelligence. However, current evaluation paradigms often treat benchmarks as isolated "silos," offering only monolithic aggregate scores that neglect the intricacies of both academic specialization and cognitive depth. This result-oriented approach fails to distinguish whether model errors stem from insufficient domain knowledge or deficiencies in cognitive capacity, thereby limiting the diagnostic value. To address this, we propose STEMVerse, a diagnostic framework designed to systematically analyze the STEM reasoning capabilities of LLMs. This framework characterizes model performance across academic specialization and cognitive complexity to map the capability required for reasoning. We re-aggregate over 20,000 STEM problems from mainstream benchmarks into a unified "Discipline $\times$ Cognition" capability space, assigning dual-axis labels to every instance. Utilizing this unified diagnostic framework, we systematically evaluate representative LLM families across varying parameter scales and training paradigms. Our empirical results reveal structural failure patterns in STEM reasoning. By integrating multi-disciplinary coverage and fine-grained cognitive stratification into a unified framework, STEMVerse provides a clear and actionable perspective for understanding the scientific reasoning characteristics of LLMs.

Figures (7)

• Figure 1: Paradigm shift from result-oriented ranking to capability-driven diagnostics.Left: Traditional benchmarks treat disciplines as isolated silos, offering only monolithic accuracy scores that neglect the intricacies and profundities of both academic specialization and cognitive depth. Right: STEMVerse restructures evaluation into a dual-axis capability matrix to pinpoint "logical blind spots."
• Figure 2: Overview of the STEMVerse. STEMVerse restructures traditional STEM benchmarks into a dual-axis capability matrix, mapping academic specializations against Bloom’s cognitive taxonomy to provide a granular, "spectral" characterization of model reasoning.
• Figure 3: Distribution of academic specializations. The composition of fine-grained academic specializations across Mathematics, Physics, Chemistry, and Biology, ensures a balanced and comprehensive coverage of the STEM knowledge landscape.
• Figure 4: Distribution of cognitive levels across disciplines. The stacked bar chart shows the percentage of problems categorized under each level of Bloom’s Taxonomy for Biology, Physics, Mathematics, and Chemistry, highlighting the benchmark's focus on high-order reasoning evaluation.
• Figure 5: Performance across academic specializations. These radar charts visualize the accuracy of the Qwen and Llama families across fine-grained specializations in Math, Biology, Physics, and Chemistry, illustrating the performance variances. For detailed results, please refer to Appendix \ref{['appendix:academic']}.