Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EasyMath: A 0-shot Math Benchmark for SLMs

Drishya Karki, Michiel Kamphuis, Angelecia Frey

TL;DR

EasyMath introduces a lightweight, open-ended benchmark for practical mathematical reasoning in small language models, covering 13 categories and evaluating 23 models (14M–4B parameters) via exact, numerical, and symbolic checks in a zero-shot setup. The study demonstrates that model size and training enhance performance, while chain-of-thought prompting yields modest gains and consistency improves with scale, validated through rigorous statistical analyses. By comparing with existing benchmarks and detailing evaluation pipelines that include symbolic verification with SymPy, EasyMath offers a realistic, robust metric for advancing on-device and resource-constrained math reasoning. The work also proposes strategies to improve performance without expanding parameters, such as targeted fine-tuning on math datasets and careful prompting, and outlines future directions including dynamic problem sets and hybrid symbolic-neural solvers.

Abstract

EasyMath is a compact benchmark for practical math reasoning in small language models. It covers thirteen categories, from basic arithmetic and order of operations to word problems, algebraic expressions, edge cases, and omits specialist topics. We tested 23 models (14M to 4B parameters) using exact, numerical, and symbolic checks on free-form answers in a zero-shot setting. Accuracy rises with size and training, chain-of-thought adds modest gains, and consistency improves at scale.

EasyMath: A 0-shot Math Benchmark for SLMs

TL;DR

EasyMath introduces a lightweight, open-ended benchmark for practical mathematical reasoning in small language models, covering 13 categories and evaluating 23 models (14M–4B parameters) via exact, numerical, and symbolic checks in a zero-shot setup. The study demonstrates that model size and training enhance performance, while chain-of-thought prompting yields modest gains and consistency improves with scale, validated through rigorous statistical analyses. By comparing with existing benchmarks and detailing evaluation pipelines that include symbolic verification with SymPy, EasyMath offers a realistic, robust metric for advancing on-device and resource-constrained math reasoning. The work also proposes strategies to improve performance without expanding parameters, such as targeted fine-tuning on math datasets and careful prompting, and outlines future directions including dynamic problem sets and hybrid symbolic-neural solvers.

Abstract

EasyMath is a compact benchmark for practical math reasoning in small language models. It covers thirteen categories, from basic arithmetic and order of operations to word problems, algebraic expressions, edge cases, and omits specialist topics. We tested 23 models (14M to 4B parameters) using exact, numerical, and symbolic checks on free-form answers in a zero-shot setting. Accuracy rises with size and training, chain-of-thought adds modest gains, and consistency improves at scale.

Paper Structure

This paper contains 34 sections, 4 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Progress in Small Language Models
  4. Advances in Mathematical Benchmarks
  5. Bridging the Gap: From Benchmarks to Real-World Applications
  6. Reflection on Current Math Benchmarks
  7. Challenges with GSM8K
  8. Limitations of Multiple-Choice Frameworks in MathQA
  9. Gaps in Hendrycks-MATH and Math-500
  10. EasyMath benchmark
  11. Methodology
  12. Categories
  13. Dataset Development and Curation
  14. Evaluation
  15. Confidence interval of intended answer
  16. ...and 19 more sections

Figures (9)

  • Figure 1: SmolLM2 models accuracy
  • Figure 2: Distribution of Questions Across Categories
  • Figure 3: Flowchart of the EasyMath benchmark evaluation process
  • Figure 4: Average accuracy vs. parameter count (log scale) across different model families
  • Figure 5: Model consistency comparison
  • ...and 4 more figures