Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can LLMs subtract numbers?

Mayank Jobanputra, Nils Philipp Walter, Maitrey Mehta, Blerta Veseli, Evan Parker Kelly Chapple, Yifan Wang, Sneha Chetani, Ellie Pavlick, Antonio Vergari, Vera Demberg

TL;DR

The paper systematically evaluates subtraction in eight pretrained LLMs across four families, revealing that subtraction accuracy substantially trails addition and exhibits a strong bias when $a<b$, with the negative sign frequently omitted. Through data generation within tokenizer ranges and varied prompting, the study uncovers that LLMs internally encode when results are negative, yet generation often fails to surface the sign, a gap bridged by instruction-tuned models which achieve near-perfect subtraction performance. Few-shot prompting yields modest, inconsistent gains, highlighting the limits of in-context learning for this task. The findings emphasize subtraction as a valuable diagnostic for numerical reasoning and suggest that instruction tuning is a more robust remedy than few-shot learning for enabling correct negative results.

Abstract

We present a systematic study of subtraction in large language models (LLMs). While prior benchmarks emphasize addition and multiplication, subtraction has received comparatively little attention despite being structurally distinct as a non-commutative operation. We evaluate eight pretrained LLMs spanning four families on addition and subtraction problems. Our experiments reveal that subtraction accuracy lags behind addition by a wide margin. We find that the errors for ($a-b$) are concentrated in cases where ($a<b$). In such cases, LLMs frequently produce the correct magnitude but omit the negative sign. Probing analyses show that LLMs internally encode whether results should be negative, yet this information is often not reflected in generated outputs. We further test well-known techniques such as few-shot learning and instruction-tuning to see if they can improve the LLMs' performance. Our results suggest that while few-shot prompting yields modest gains, the instruction-tuned models achieve near-perfect accuracies in generating the negative sign. Together, these findings provide a clearer characterization of the limitations and recoverability of LLMs' arithmetic capabilities in subtraction.

Can LLMs subtract numbers?

TL;DR

The paper systematically evaluates subtraction in eight pretrained LLMs across four families, revealing that subtraction accuracy substantially trails addition and exhibits a strong bias when , with the negative sign frequently omitted. Through data generation within tokenizer ranges and varied prompting, the study uncovers that LLMs internally encode when results are negative, yet generation often fails to surface the sign, a gap bridged by instruction-tuned models which achieve near-perfect subtraction performance. Few-shot prompting yields modest, inconsistent gains, highlighting the limits of in-context learning for this task. The findings emphasize subtraction as a valuable diagnostic for numerical reasoning and suggest that instruction tuning is a more robust remedy than few-shot learning for enabling correct negative results.

Abstract

We present a systematic study of subtraction in large language models (LLMs). While prior benchmarks emphasize addition and multiplication, subtraction has received comparatively little attention despite being structurally distinct as a non-commutative operation. We evaluate eight pretrained LLMs spanning four families on addition and subtraction problems. Our experiments reveal that subtraction accuracy lags behind addition by a wide margin. We find that the errors for () are concentrated in cases where (). In such cases, LLMs frequently produce the correct magnitude but omit the negative sign. Probing analyses show that LLMs internally encode whether results should be negative, yet this information is often not reflected in generated outputs. We further test well-known techniques such as few-shot learning and instruction-tuning to see if they can improve the LLMs' performance. Our results suggest that while few-shot prompting yields modest gains, the instruction-tuned models achieve near-perfect accuracies in generating the negative sign. Together, these findings provide a clearer characterization of the limitations and recoverability of LLMs' arithmetic capabilities in subtraction.

Paper Structure

This paper contains 24 sections, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Experimental Setup
  3. Data generation
  4. Sampling operands $a$ and $b$.
  5. Prompt Variants.
  6. Zero-shot vs. n-shot.
  7. LLM Inference
  8. Results
  9. Can LLMs subtract numbers?
  10. Subtraction with $a>b$ versus $a<b$
  11. What errors do LLMs make when $a<b$?
  12. Did LLMs simply forget to include the sign?
  13. Can we make LLMs better for a<b?
  14. In-context learning with pretrained LLMs
  15. Instruction-tuned LLMs
  16. ...and 9 more sections

Figures (9)

  • Figure 1: Zero-shot performance of LLMs on addition and subtraction problems (averaged across prompt variants). Subtraction is consistently harder, even for LLMs that perform well on addition.
  • Figure 2: Zero-shot performance on subtraction ($a-b$) across operand and prompt variants. Pretrained LLMs perform subtraction well when $a>b$ but fail almost completely when $a<b$, showing a strong asymmetry.
  • Figure 3: Zero-shot performance on the -b+a input pairs. This plot shows the same asymmetry as Figure \ref{['fig:q_2']}.
  • Figure 4: Zero-shot performance on a-b (above) and -b+a (below). The gap between the accuracy with and without the '-' shows that pretrained LLMs often compute the correct magnitude but omit the negative sign.
  • Figure 5: Zero-shot performance on multi-token addition and subtraction problems (averaged across prompt variants). Similar to single-token subtraction, multi-token subtraction is also consistently harder for all the LLMs.
  • ...and 4 more figures