Towards Reasoning Ability of Small Language Models

TL;DR

ThinkSLM provides the first extensive benchmark to systematically evaluate reasoning in small language models by comparing trained-from-scratch, quantized, pruned, and distilled variants across 17 reasoning tasks. The framework demonstrates that training data quality and alignment strategies often outweigh model size, with quantization preserving reasoning while pruning degrades it significantly. The findings reveal nuanced robustness patterns and caution against overreliance on scale, suggesting a practical path to strong SLM reasoning through structured training and compression. The publicly available ThinkSLM leaderboard offers a benchmark for ongoing progress in SLM reasoning capabilities.

Abstract

Reasoning has long been viewed as an emergent property of large language models (LLMs). However, recent studies challenge this assumption, showing that small language models (SLMs) can also achieve competitive reasoning performance. This paper introduces ThinkSLM, the first extensive benchmark to systematically evaluate and study the reasoning abilities of SLMs trained from scratch or derived from LLMs through quantization, pruning, and distillation. We first establish a reliable evaluation criterion comparing available methods and LLM judges against our human evaluations. Then we present a study evaluating 72 diverse SLMs from six major model families across 17 reasoning benchmarks. We repeat all our experiments three times to ensure a robust assessment. Our findings show that: 1) reasoning ability in SLMs is strongly influenced by training methods and data quality rather than solely model scale; 2) quantization preserves reasoning capability, while pruning significantly disrupts it; 3) larger models consistently exhibit higher robustness against adversarial perturbations and intermediate reasoning, but certain smaller models closely match or exceed the larger models' performance. Our findings challenge the assumption that scaling is the only way to achieve strong reasoning. Instead, we foresee a future where SLMs with strong reasoning capabilities can be developed through structured training or post-training compression. Our ThinkSLM Leaderboard is publicly available at: https://ctrl-gaurav.github.io/thinkslm.github.io/

Figures (16)

• Figure 1: (a) Overall performance on 4 common reasoning benchmarks. (b) Performance on Sorting Tasks. The x-axis represents different models (with parameters in billions), the y-axis represents the mean accuracy, and the bar represents variance (3-folds). Each line corresponds to different sorting tasks (8 +ve, 8 mixed, 16 +ve, 16 mixed, 32 +ve, and 32 mixed numbers). (c) Effect of Prompts on SLM Performance on the GSM8K. Each line corresponds to different prompting strategies (Direct I/O, Chain-of-Thought (CoT), 5-shot, 5-shot CoT, and 8-shot).
• Figure 2: Impact of Quantization on Model Performance across Different Benchmarks. The figure shows the performance of different models on GSM8K (Direct I/O), Average of ARC-E, and CommonsenseQA, and Average of all sorting tasks with varying quantization levels. All results are from Qwen2.5 Family. The x-axis represents the parameters size (in billions), and the y-axis represents the mean accuracy and bar represents variance (3-folds).
• Figure 3: Performance of different models on GSM8K (Direct I/O), ARC, CommonsenseQA, and sorting tasks. The x-axis represents the parameters size (in billions), and the y-axis represents the mean accuracy, with error bars indicating the variance (3-folds).
• Figure 4: Impact of Quantization on Model Performance across Different Benchmarks.
• Figure 5: Impact of Quantization on Model Performance across Sorting Tasks.