Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results

TL;DR

The paper scrutinizes cross-lingual mathematical reasoning in LLMs using MGSM, revealing that apparent language gaps were largely artifacts of translation errors and inconsistent answer extraction. By applying semi-automatic data cleaning, correcting translations, and adopting more robust answer parsing (including handling Bengali numerals), the authors demonstrate that the previously observed multilingual gap can disappear for strong models. The study emphasizes the critical importance of data quality and evaluation detail in multilingual benchmarks and releases a corrected MGSM dataset to the community. Overall, the work cautions against over-interpreting multilingual performance without addressing data and methodology biases, especially in high-headroom benchmarks.

Abstract

Most current large language models (LLMs) support a wide variety of languages in addition to English, including high-resource languages (e.g. German, Chinese, French), as well as low-resource ones (e.g. Swahili, Telugu). In addition they have also shown impressive capabilities in different domains, like coding, science and math. In this short paper, taking math as an example domain, we study the performance of different LLMs across languages. Experimental results show that there exists a non-negligible and consistent gap in the performance of the models across languages. Interestingly, and somewhat against expectations, the gap exists for both high- and low-resource languages. We hope that these results influence further research into cross-lingual capability generalization for next generation LLMs. If it weren't for the fact that they are false! By analyzing one of the standard multilingual math benchmarks (MGSM), we determine that several translation errors are present in the data. Furthermore, the lack of standardized answer extraction from LLM outputs further influences the final results. We propose a method for automatic quality assurance to address the first issue at scale, and give recommendations to address the second one. Combining these two approaches we show that the aforementioned language gap mostly disappears, leading to completely different conclusions from our research. We additionally release the corrected dataset to the community.

Figures (4)

• Figure 1: Number of modified problems and categorization of changes. The total number of questions in each languages is 250.
• Figure 2: Function to extract answer by Kaggle
• Figure 3: Function to last number in response
• Figure 4: The prompt used to classify modifications to MGSM questions.