AI4Math: A Native Spanish Benchmark for University-Level Mathematical Reasoning in Large Language Models

TL;DR

AI4Math introduces a native Spanish benchmark for university-level mathematical reasoning, addressing translation-related semantic drift in existing English-centric evaluations. It comprises 105 original problems across Algebra, Calculus, Geometry, Probability, Number Theory, Combinatorics, and Mathematical Logic, each with a final answer and a detailed human solution. Evaluations of six large language models under four settings (Spanish/English, zero-shot and zero-shot chain-of-thought) reveal that compact/open models like o3-mini and DeepSeek-R1/V3 can surpass 70% accuracy, while GPT-4o mini and LLaMA 3.3 70B lag, with minimal language gaps except for occasional Spanish advantages. The results underscore native-language benchmarks as a diagnostic tool for uncovering domain-specific reasoning weaknesses and stress the value of community-driven benchmark development, while highlighting areas—Geometry, Combinatorics, Probability—where current systems struggle and pointing to future work in expanding the dataset and incorporating new models and modalities.

Abstract

Existing mathematical reasoning benchmarks are predominantly English only or translation-based, which can introduce semantic drift and mask languagespecific reasoning errors. To address this, we present AI4Math, a benchmark of 105 original university level math problems natively authored in Spanish. The dataset spans seven advanced domains (Algebra, Calculus, Geometry, Probability, Number Theory, Combinatorics, and Logic), and each problem is accompanied by a step by step human solution. We evaluate six large language models GPT 4o, GPT 4o mini, o3 mini, LLaMA 3.3 70B, DeepSeek R1 685B, and DeepSeek V3 685B under four configurations: zero shot and chain of thought, each in Spanish and English. The top models (o3 mini, DeepSeek R1 685B, DeepSeek V3 685B) achieve over 70% accuracy, whereas LLaMA 3.3 70B and GPT-4o mini remain below 40%. Most models show no significant performance drop between languages, with GPT 4o even performing better on Spanish problems in the zero shot setting. Geometry, Combinatorics, and Probability questions remain persistently challenging for all models. These results highlight the need for native-language benchmarks and domain-specific evaluations to reveal reasoning failures not captured by standard metrics.

Figures (5)

• Figure 1: Overall success rate per model. The figure presents the model names on the X-axis (o3-mini, GPT-4o, GPT-4o mini, DeepSeek-R1 685B, DeepSeek-V3 685B, LLaMA 3.3 70B) and the percentage of successfully solved problems on the Y-axis. For each model, the success rate is categorized by evaluation settings (Zero Shot and Zero Shot - Chain of Thought) and by language (Spanish and English). The comparison highlights performance differences across models and evaluation configurations, illustrating how each model's success rate varies between languages and reasoning approaches.
• Figure 2: The figure presents the percentage of correctly solved problems by model and domain under the English Zero-Shot configuration. Each marker represents the performance of a specific model across seven mathematical domains: Combinatorics, Calculus, Geometry, Mathematical Logic, Probability, Number Theory, and Algebra. Tab.\ref{['Table:domainChiSqauredEnglishZS']} and Tab. \ref{['Tab:domainChiSqauredEnglishZSModels']} report the statistical differences observed across models and mathematical domains.
• Figure 3: Presents the percentage of correctly solved problems by model and domain under the English Zero-Shot Chain of Thought configuration. Each marker represents the performance of a specific model across seven mathematical domains: Combinatorics, Calculus, Geometry, Mathematical Logic, Probability, Number Theory, and Algebra. Tab.\ref{['Table:domainChiSqauredEnglishCoT']} and Tab. \ref{['tab:domainChiSqauredEnglishCoTModels']} report the statistical differences observed across models and mathematical domains.
• Figure 4: presents the success rate of large language models across different mathematical domains under the Spanish Zero-Shot configuration. Each marker corresponds to a domain Combinatorics, Calculus, Geometry, Mathematical Logic, Probability, Number Theory, Algebra and shows the percentage of correctly solved problems by each model. Tab \ref{['Tab:domainChiSqauredSpanishZS']} and Tab. \ref{['fig:domainChiSqauredSpanishZSModels']} report the statistical differences observed across models and mathematical domains.
• Figure 5: The figure presents the success rate of large language models across different mathematical domains under the Spanish Zero-Shot Chain of Thought configuration. Each marker corresponds to a domain Combinatorics, Calculus, Geometry, Mathematical Logic, Probability, Number Theory, Algebra and shows the percentage of correctly solved problems by each model. Tab \ref{['Table:domainChiSqauredSpanishCoTModels']} and Tab. \ref{['Tab:domainChiSquaredSpanishCoTdomain']} report the statistical differences observed across models and mathematical domains.