Investigating Numerical Translation with Large Language Models

TL;DR

This work investigates the reliability of LLM-based translation for numerical data, with emphasis on large-unit conversions. It introduces a ten-type EN-ZH numerical translation dataset derived from real business data and evaluates a range of generic and translation-oriented LLMs using a Pass Rate metric. It finds that no model consistently handles all numerical types, with large-unit translation being the hardest, but demonstrates three mitigation strategies—ICL, COT, and post-editing—where post-editing yields the strongest gains. The results offer a practical benchmark and a viable post-processing approach to improve numerical translation in real-world applications like finance and healthcare, guiding future research on numerically robust LLM systems.

Abstract

The inaccurate translation of numbers can lead to significant security issues, ranging from financial setbacks to medical inaccuracies. While large language models (LLMs) have made significant advancements in machine translation, their capacity for translating numbers has not been thoroughly explored. This study focuses on evaluating the reliability of LLM-based machine translation systems when handling numerical data. In order to systematically test the numerical translation capabilities of currently open source LLMs, we have constructed a numerical translation dataset between Chinese and English based on real business data, encompassing ten types of numerical translation. Experiments on the dataset indicate that errors in numerical translation are a common issue, with most open-source LLMs faltering when faced with our test scenarios. Especially when it comes to numerical types involving large units like ``million", ``billion", and "yi", even the latest llama3.1 8b model can have error rates as high as 20%. Finally, we introduce three potential strategies to mitigate the numerical mistranslations for large units.

Figures (1)

• Figure 1: The framework of PE strategy.