Injecting Numerical Reasoning Skills into Language Models

TL;DR

This work demonstrates that numerical reasoning can be effectively injected into pre-trained language models by automatically generating large-scale numerical and textual synthetic data and training a simple encoder-decoder architecture end-to-end. The GenBERT model, initialized from BERT and trained with ND and TD data plus MLM, achieves competitive DROP performance without specialized arithmetic modules, while also generalizing to math word problems and preserving standard reading comprehension. The approach provides a general recipe for augmenting LMs with new skills whenever automatic data generation is feasible, and the resulting weights transfer to other architectures, broadening applicability. Overall, the paper shows that skill augmentation via synthetic data is a practical pathway to more capable, flexible language models.

Abstract

Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only. Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility. In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup. We show that pre-training our model, GenBERT, on this data, dramatically improves performance on DROP (49.3 $\rightarrow$ 72.3 F1), reaching performance that matches state-of-the-art models of comparable size, while using a simple and general-purpose encoder-decoder architecture. Moreover, GenBERT generalizes well to math word problem datasets, while maintaining high performance on standard RC tasks. Our approach provides a general recipe for injecting skills into large pre-trained LMs, whenever the skill is amenable to automatic data augmentation.

Figures (5)

• Figure 1: An overview of our approach for injecting numerical skills into a pre-trained LM. (a) We add two pre-training steps over large amounts of synthetic numerical data (ND) and textual data (TD); (b) we further fine-tune the model over either numerical reasoning datasets (DROP, MAWPS) or reading comprehension datasets (SQuAD).
• Figure 2: GenBERT's network architecture: (a) a high-level overview of the network, including a generative head (red), two span-extraction heads (yellow), and an answer type head. (b) a closer overview of GenBERT's generative head.
• Figure 3: Template extraction and instantiation. A template (in red) is extracted from a MWP sentence, using categories for containers, entities, verbs, attributes and numbers, according to hosseini2014learning. For generation, the categories are instantiated with a domain-specific vocabulary.
• Figure 4: Progression of eval accuracy (EM) of GenBERT, for different pre-training settings listed in §\ref{['section:pretraining_results']}.
• Figure 5: Breakdown of model accuracy (EM) by the number of terms in the arithmetic expression, for the MWP datasets AddSub, SOp and SEq.