Do NLP Models Know Numbers? Probing Numeracy in Embeddings

TL;DR

This work reveals that numeracy is an emergent property of common word embeddings, not solely a trained arithmetic capability. By analyzing the DROP dataset with NAQANet and by constructing synthetic probing tasks, the authors show that magnitude information is present across pre-trained embeddings, with character-level representations offering the strongest numeracy. However, neural models struggle to extrapolate numeracy beyond training ranges, though data augmentation can mitigate some of these failures. The findings highlight both the potential and the limits of current embeddings for numerical reasoning in NLP, with implications for robust, numeracy-aware language understanding systems.

Abstract

The ability to understand and work with numbers (numeracy) is critical for many complex reasoning tasks. Currently, most NLP models treat numbers in text in the same way as other tokens---they embed them as distributed vectors. Is this enough to capture numeracy? We begin by investigating the numerical reasoning capabilities of a state-of-the-art question answering model on the DROP dataset. We find this model excels on questions that require numerical reasoning, i.e., it already captures numeracy. To understand how this capability emerges, we probe token embedding methods (e.g., BERT, GloVe) on synthetic list maximum, number decoding, and addition tasks. A surprising degree of numeracy is naturally present in standard embeddings. For example, GloVe and word2vec accurately encode magnitude for numbers up to 1,000. Furthermore, character-level embeddings are even more precise---ELMo captures numeracy the best for all pre-trained methods---but BERT, which uses sub-word units, is less exact.

Figures (3)

• Figure 1: We train a probing model to decode a number from its word embedding over a random 80% of the integers from [-500, 500], e.g., "71" $\to$ 71.0. We plot the model's predictions for all numbers from [-2000, 2000]. The model accurately decodes numbers within the training range (in blue), i.e., pre-trained embeddings like GloVe and BERT capture numeracy. However, the probe fails to extrapolate to larger numbers (in red). The Char-CNN (e) and Char-LSTM (f) are trained jointly with the probing model.
• Figure 2: Three DROP questions that require numerical reasoning; the state-of-the-art NAQANet answers every question correct. Plausible answer candidates to the questions are underlined and the model's predictions are shown in bold.
• Figure 3: Our probing setup. We pass numbers through a pre-trained embedder (e.g., BERT, GloVe) and train a probing model to solve numerical tasks such as finding a list's maximum, decoding a number, or adding two numbers. If the probing model generalizes to held-out numbers, the pre-trained embeddings must contain numerical information. We provide numbers as either words (shown here), digits ("9"), floats ("9.1"), or negatives ("-9").