Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks

Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Chitta Baral

TL;DR

This work addresses the need for a format-independent numerical reasoning benchmark by introducing NUMBERGAME, which spans eight question formats and includes four novel data types alongside existing datasets. The authors define prerequisites for practical reasoning, including explicit format detection, a common intermediate representation, commonsense knowledge, and strategies for data imbalance, and they evaluate multiple baselines including a knowledge-hunting approach atop extended NumNet+V2. Empirical results show substantial gaps between current baselines and human performance, highlighting the challenge of cross-format numerical reasoning and the value of external knowledge integration. The study advances the field by proposing a multifaceted, hard-to-solve benchmark that motivates development of general-purpose, neuro-symbolic reasoning systems and informs future directions in transfer learning and robust multi-task reasoning.

Abstract

Numerical reasoning is often important to accurately understand the world. Recently, several format-specific datasets have been proposed, such as numerical reasoning in the settings of Natural Language Inference (NLI), Reading Comprehension (RC), and Question Answering (QA). Several format-specific models and architectures in response to those datasets have also been proposed. However, there exists a strong need for a benchmark which can evaluate the abilities of models, in performing question format independent numerical reasoning, as (i) the numerical reasoning capabilities we want to teach are not controlled by question formats, (ii) for numerical reasoning technology to have the best possible application, it must be able to process language and reason in a way that is not exclusive to a single format, task, dataset or domain. In pursuit of this goal, we introduce NUMBERGAME, a multifaceted benchmark to evaluate model performance across numerical reasoning tasks of eight diverse formats. We add four existing question types in our compilation. Two of the new types we add are about questions that require external numerical knowledge, commonsense knowledge and domain knowledge. For building a more practical numerical reasoning system, NUMBERGAME demands four capabilities beyond numerical reasoning: (i) detecting question format directly from data (ii) finding intermediate common format to which every format can be converted (iii) incorporating commonsense knowledge (iv) handling data imbalance across formats. We build several baselines, including a new model based on knowledge hunting using a cheatsheet. However, all baselines perform poorly in contrast to the human baselines, indicating the hardness of our benchmark. Our work takes forward the recent progress in generic system development, demonstrating the scope of these under-explored tasks.

Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks

TL;DR

This work addresses the need for a format-independent numerical reasoning benchmark by introducing NUMBERGAME, which spans eight question formats and includes four novel data types alongside existing datasets. The authors define prerequisites for practical reasoning, including explicit format detection, a common intermediate representation, commonsense knowledge, and strategies for data imbalance, and they evaluate multiple baselines including a knowledge-hunting approach atop extended NumNet+V2. Empirical results show substantial gaps between current baselines and human performance, highlighting the challenge of cross-format numerical reasoning and the value of external knowledge integration. The study advances the field by proposing a multifaceted, hard-to-solve benchmark that motivates development of general-purpose, neuro-symbolic reasoning systems and informs future directions in transfer learning and robust multi-task reasoning.

Abstract

Numerical reasoning is often important to accurately understand the world. Recently, several format-specific datasets have been proposed, such as numerical reasoning in the settings of Natural Language Inference (NLI), Reading Comprehension (RC), and Question Answering (QA). Several format-specific models and architectures in response to those datasets have also been proposed. However, there exists a strong need for a benchmark which can evaluate the abilities of models, in performing question format independent numerical reasoning, as (i) the numerical reasoning capabilities we want to teach are not controlled by question formats, (ii) for numerical reasoning technology to have the best possible application, it must be able to process language and reason in a way that is not exclusive to a single format, task, dataset or domain. In pursuit of this goal, we introduce NUMBERGAME, a multifaceted benchmark to evaluate model performance across numerical reasoning tasks of eight diverse formats. We add four existing question types in our compilation. Two of the new types we add are about questions that require external numerical knowledge, commonsense knowledge and domain knowledge. For building a more practical numerical reasoning system, NUMBERGAME demands four capabilities beyond numerical reasoning: (i) detecting question format directly from data (ii) finding intermediate common format to which every format can be converted (iii) incorporating commonsense knowledge (iv) handling data imbalance across formats. We build several baselines, including a new model based on knowledge hunting using a cheatsheet. However, all baselines perform poorly in contrast to the human baselines, indicating the hardness of our benchmark. Our work takes forward the recent progress in generic system development, demonstrating the scope of these under-explored tasks.

Paper Structure

This paper contains 36 sections, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Data Creation
  4. Novel Datasets
  5. Daily Life Maths Requiring Numerical Common Sense Knowledge:
  6. Application of Maths requiring domain knowledge:
  7. Quantitative Comparison Requiring Common Sense Knowledge along with Numerical Common Sense:
  8. Completion type questions
  9. Collection of Existing Datasets
  10. Reading Comprehension (RC) with Explicit Math
  11. Reading Comprehension involving Implicit Math
  12. Quantitative NLI
  13. Arithmetic Word Problems
  14. Combining all types of data
  15. Partitioning the dataset:
  16. ...and 21 more sections

Figures (7)

  • Figure 1: Baseline Model overview depicting the various questions types and their answers.
  • Figure 2: Distribution of average vocabulary in the data set.
  • Figure 3: Distribution of Part of Speech tags across question types.
  • Figure 4: Knowledge Hunting augmented Model overview depicting the various questions types and their answers.
  • Figure 5: Conversion of various question types to reading comprehension format
  • ...and 2 more figures