Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Effect of Scripts and Formats on LLM Numeracy

Varshini Reddy, Craig W. Schmidt, Seth Ebner, Adam Wiemerslage, Yuval Pinter, Chris Tanner

TL;DR

This work investigates how numeral scripts and formatting influence large language model numeracy beyond underlying arithmetic, revealing a substantial script tax when non-H Hindu–Arabic representations are used. By constructing two controlled datasets—the Multiscript Numeral Dataset and the Format-Variation Numeral Dataset—and evaluating nine diverse LLMs, the study dissects script recognition, translation, and arithmetic computation under representational shifts. Key findings show that non-HA scripts cause large accuracy drops in arithmetic (approximately 66–87% on average), but prompting strategies, especially explicit mapping between scripts and native-language prompts, can dramatically reduce these gaps; few-shot prompting yields particularly strong improvements. The results underscore the fragility of numerical reasoning under representational variation and offer actionable prompting techniques to improve robustness when handling diverse numeral scripts and formatting in real-world applications.

Abstract

Large language models (LLMs) have achieved impressive proficiency in basic arithmetic, rivaling human-level performance on standard numerical tasks. However, little attention has been given to how these models perform when numerical expressions deviate from the prevailing conventions present in their training corpora. In this work, we investigate numerical reasoning across a wide range of numeral scripts and formats. We show that LLM accuracy drops substantially when numerical inputs are rendered in underrepresented scripts or formats, despite the underlying mathematical reasoning being identical. We further demonstrate that targeted prompting strategies, such as few-shot prompting and explicit numeral mapping, can greatly narrow this gap. Our findings highlight an overlooked challenge in multilingual numerical reasoning and provide actionable insights for working with LLMs to reliably interpret, manipulate, and generate numbers across diverse numeral scripts and formatting styles.

The Effect of Scripts and Formats on LLM Numeracy

TL;DR

This work investigates how numeral scripts and formatting influence large language model numeracy beyond underlying arithmetic, revealing a substantial script tax when non-H Hindu–Arabic representations are used. By constructing two controlled datasets—the Multiscript Numeral Dataset and the Format-Variation Numeral Dataset—and evaluating nine diverse LLMs, the study dissects script recognition, translation, and arithmetic computation under representational shifts. Key findings show that non-HA scripts cause large accuracy drops in arithmetic (approximately 66–87% on average), but prompting strategies, especially explicit mapping between scripts and native-language prompts, can dramatically reduce these gaps; few-shot prompting yields particularly strong improvements. The results underscore the fragility of numerical reasoning under representational variation and offer actionable prompting techniques to improve robustness when handling diverse numeral scripts and formatting in real-world applications.

Abstract

Large language models (LLMs) have achieved impressive proficiency in basic arithmetic, rivaling human-level performance on standard numerical tasks. However, little attention has been given to how these models perform when numerical expressions deviate from the prevailing conventions present in their training corpora. In this work, we investigate numerical reasoning across a wide range of numeral scripts and formats. We show that LLM accuracy drops substantially when numerical inputs are rendered in underrepresented scripts or formats, despite the underlying mathematical reasoning being identical. We further demonstrate that targeted prompting strategies, such as few-shot prompting and explicit numeral mapping, can greatly narrow this gap. Our findings highlight an overlooked challenge in multilingual numerical reasoning and provide actionable insights for working with LLMs to reliably interpret, manipulate, and generate numbers across diverse numeral scripts and formatting styles.
Paper Structure (33 sections, 1 equation, 8 figures, 10 tables)

This paper contains 33 sections, 1 equation, 8 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Multiscript Numeral Dataset
  4. Format-Variation Numeral Dataset
  5. Multiscript Numerical Reasoning
  6. Script Identification and Translation
  7. Arithmetic Computation
  8. Impact of Numeral Script Variation on Arithmetic Performance
  9. Bridging the Performance Gap through Prompting Strategies
  10. Quantifying Factors Affecting Multiscript Arithmetic Performance
  11. Influence of Operation and Tokenization
  12. Impact of Numeral Scripts
  13. Prompt Engineering Effects
  14. Arithmetic Reasoning with Formatted Numbers
  15. The Baseline
  16. ...and 18 more sections

Figures (8)

  • Figure 1: LLM numerical reasoning is sensitive to the script and format used; illustrative samples of inputs and outputs from Llama 3.3.
  • Figure 2: Comparison of model accuracy on the arithmetic task. Error bars represent the standard deviation and the dots represent the maximum accuracy among non-HA scripts.
  • Figure 3: Comparison of model accuracy on the arithmetic task. Error bars represent the standard deviation and and the dots represent the maximum accuracy among non-HA scripts
  • Figure 4: Comparison of model accuracy on the arithmetic task across formatted variants. Error bars represent the standard deviation.
  • Figure 5: Comparison of model accuracy on the arithmetic task. Error bars denote standard deviation, and dots indicate the maximum accuracy across formats. For F1, the bar shows the average over the four prompting strategies, while the dot represents the best-performing prompt.
  • ...and 3 more figures