NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks

TL;DR

NumGLUE introduces a multi-task benchmark of eight arithmetic-focused tasks totaling ~100K questions to probe robust numerical reasoning in language. It combines four novel datasets with four existing ones and shows that current SOTA models struggle compared to humans, even with fine-tuning, indicating a fundamental barrier to generalized arithmetic understanding. The authors propose a memory-augmented Ex-NumNet with external knowledge retrieval (MATH KB/IR) and demonstrate a 3.4% average gain from multi-task joint training over task-specific models, highlighting transfer across related arithmetic tasks. This work establishes a challenging, cross-task platform that encourages robust numeric reasoning within language and points toward broader mathematical reasoning capabilities.

Abstract

Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems. While many datasets and models have been developed to this end, state-of-the-art AI systems are brittle; failing to perform the underlying mathematical reasoning when they appear in a slightly different scenario. Drawing inspiration from GLUE that was proposed in the context of natural language understanding, we propose NumGLUE, a multi-task benchmark that evaluates the performance of AI systems on eight different tasks, that at their core require simple arithmetic understanding. We show that this benchmark is far from being solved with neural models including state-of-the-art large-scale language models performing significantly worse than humans (lower by 46.4%). Further, NumGLUE promotes sharing knowledge across tasks, especially those with limited training data as evidenced by the superior performance (average gain of 3.4% on each task) when a model is jointly trained on all the tasks as opposed to task-specific modeling. Finally, we hope that NumGLUE will encourage systems that perform robust and general arithmetic reasoning within language, a first step towards being able to perform more complex mathematical reasoning.

Figures (14)

• Figure 1: A system that can robustly perform numeric reasoning over language should be able to solve problems such as the above, regardless of how the problem is posed. However, we observe existing systems are brittle; producing inconsistent solutions to such minor stylistic variations.
• Figure 2: Performance of zeroshot, fewshot and finetuning baselines (Section \ref{['exp']}) across NumGLUE. There is a signficant gap between the highest performing model and the human baseline. ZS: Zeroshot, GPT3I: GPT3-Instruct, MT: Multi-task, TS: Task-specific, QO: Question Only, CO: Context Only, EXNN: Ex-NumNet,FS: Few-shot, OS: Oversampling, IR: Information Retrieval, CIR: Conditional Information Retrieval.
• Figure 3: Our proposed memory-augmented model that detects the type of task (1-8), uses Information Retrieval from MATH KB and append the information that gets fed to Ex-NumNet
• Figure 4: Our dataset NumGLUE (center in the yellow circle) has been positioned with respect to existing datasets. T1-T8 represents 8 tasks. Note that, NumGLUE contains the feature of being format invariant unlike other datasets. Position of datasets within clusters is done based on their semantic category, for example T1 Numerical Commonsense QA is closer to the cluster of Commonsense Reasoning + Knowledge of Facts; its position reflects the same
• Figure 5: Step by step data creation process for task 1, 2 and 4 questions