Cross-Platform Evaluation of Reasoning Capabilities in Foundation Models

TL;DR

This study addresses how well contemporary foundation models reason across disciplines and platforms. It introduces an infrastructure-agnostic benchmark evaluated across HPC, cloud, and university clusters, expanding from 19 to 79 problems over eight domains and measuring both final answers and intermediate reasoning steps. Key findings show that training data quality and architectural choice often trump sheer parameter count, revealing a parameter-efficiency paradox and a transparency-versus-correctness trade-off in reasoning. The work provides practical guidance for selecting models for education, production, and research, and offers a reproducible, cross-platform framework for longitudinal tracking of reasoning capabilities.

Abstract

This paper presents a comprehensive cross-platform evaluation of reasoning capabilities in contemporary foundation models, establishing an infrastructure-agnostic benchmark across three computational paradigms: HPC supercomputing (MareNostrum 5), cloud platforms (Nebius AI Studio), and university clusters (a node with eight H200 GPUs). We evaluate 15 foundation models across 79 problems spanning eight academic domains (Physics, Mathematics, Chemistry, Economics, Biology, Statistics, Calculus, and Optimization) through three experimental phases: (1) Baseline establishment: Six models (Mixtral-8x7B, Phi-3, LLaMA 3.1-8B, Gemma-2-9b, Mistral-7B, OLMo-7B) evaluated on 19 problems using MareNostrum 5, establishing methodology and reference performance; (2) Infrastructure validation: The 19-problem benchmark repeated on university cluster (seven models including Falcon-Mamba state-space architecture) and Nebius AI Studio (nine state-of-the-art models: Hermes-4 70B/405B, LLaMA 3.1-405B/3.3-70B, Qwen3 30B/235B, DeepSeek-R1, GPT-OSS 20B/120B) to confirm infrastructure-agnostic reproducibility; (3) Extended evaluation: Full 79-problem assessment on both university cluster and Nebius platforms, probing generalization at scale across architectural diversity. The findings challenge conventional scaling assumptions, establish training data quality as more critical than model size, and provide actionable guidelines for model selection across educational, production, and research contexts. The tri-infrastructure methodology and 79-problem benchmark enable longitudinal tracking of reasoning capabilities as foundation models evolve.

Figures (22)

• Figure 1: Baseline model performance across difficulty levels (MareNostrum 5, 19 problems). All models exhibit monotonic performance decrease with increasing complexity, with approximately 15–25% degradation per difficulty tier.
• Figure 2: Representative example of the dataset schema. Each problem includes a Problem Statement, Final Result, and Step-by-Step Result, illustrated for three difficulty tiers (Easy, Medium, Hard) within the Physics domain. This fixed schema is used across all eight domains to standardize evaluation and facilitate process-level scoring.
• Figure 3: Comprehensive analysis of foundation model performance on university cluster infrastructure. (Top left) Overall score and step accuracy comparison across seven models, showing Phi-4-mini's dominance in both metrics. (Top center) Domain-specific performance heatmap revealing systematic patterns: Phi-4-mini excels in Chemistry (0.816) and Economics (0.833), while Optimization (bottom row) remains universally challenging across all architectures. (Top right) Consistency scores (lower is better) demonstrating Falcon-Mamba-7B's exceptional stability (0.029), followed by Phi-4-mini (0.032), critical for production deployment. (Bottom left) Performance degradation across difficulty levels shows approximately 20% drop per tier for most models, with Phi-4-mini maintaining the highest hard problem performance (0.505). (Bottom center) Cross-model domain statistics identify Economics as easiest (mean 0.745 $\pm$ 0.129) and Optimization as hardest (0.408 $\pm$ 0.089), while Calculus exhibits remarkably low variance (0.034), indicating model-independent performance. (Bottom right) Infrastructure validation comparing MareNostrum 5 supercomputer with university cluster shows minimal performance degradation (LLaMA-3.1-8B: -2.9%, Phi-3-mini: -1.1%), confirming reasoning quality is infrastructure-agnostic. Color intensity in heatmap represents score magnitude (green = high, red = low).
• Figure 4: Nebius API (19 problems): overall average final score per model. Error bars denote the mean per-evaluation standard deviation.
• Figure 5: Nebius API (19 problems): average step-accuracy by model.