SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation

Abstract

Large language models often default to step-by-step computation even when efficient numerical shortcuts are available. This raises a basic question: do they exhibit number sense in a human-like behavioral sense, i.e., the ability to recognize numerical structure, apply shortcuts when appropriate, and avoid them when they are not? We introduce SenseMath, a controlled benchmark for evaluating structure-sensitive numerical reasoning in LLMs. SenseMath contains 4,800 items spanning eight shortcut categories and four digit scales, with matched strong-shortcut, weak-shortcut, and control variants. It supports three evaluation settings of increasing cognitive demand: Shortcut Use (whether models can apply shortcuts on shortcut-amenable problems); Applicability Judgment (whether they can recognize when a shortcut is appropriate or misleading); and Problem Generation (whether they can generate new problem items that correctly admit a given type of shortcut). Our evaluation across five LLMs, ranging from GPT-4o-mini to Llama-3.1-8B, shows a consistent pattern: when explicitly prompted, models readily adopt shortcut strategies and achieve substantial accuracy gains on shortcut-amenable items (up to 15%), yet under standard chain-of-thought prompting they spontaneously employ such strategies in fewer than 40% of cases, even when they demonstrably possess the requisite capability. Moreover, this competence is confined to the Use level; models systematically over-generalise shortcuts to problems where they do not apply, and fail to generate valid shortcut-bearing problems from scratch. Together, these results suggest that current LLMs exhibit procedural shortcut fluency without the structural understanding of when and why shortcuts work that underlies human number sense.

Figures (10)

• Figure 1: SenseMath overview. Each item has matched strong-shortcut, weak-shortcut, and control variants administered under CoT and number-sense prompting. The matched design isolates selective shortcut exploitation from general prompt effects.
• Figure 2: SU rate on strong items at $d{=}4$ across three prompting conditions.
• Figure 3: SU rate under CoT increases with digit scale.
• Figure 4: Average accuracy gain from NS prompting (vs. CoT) across all digit scales, by variant.
• Figure 5: RQ2 analysis. (a) NS accuracy gain on strong items increases with digit scale for capable models; 8B models show no benefit. (b) Normalized NS improvement per category at $d{=}8$: $(\text{NS}-\text{CoT})/(1-\text{CoT})$; red dashed circle = zero baseline.