Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Translation: Evaluating Mathematical Reasoning Capabilities of LLMs in Sinhala and Tamil

Sukumar Kishanthan, Kumar Thushalika, Buddhi Jayasekara, Asela Hevapathige

TL;DR

This work tackles whether LLMs genuinely reason mathematically in Sinhala and Tamil or rely on translation to English-centric representations. It introduces a six-type taxonomy of math word problems and builds a native-language dataset authored by fluent speakers, enabling zero-shot evaluation of four prominent LLMs. Across languages, basic arithmetic reasoning transfers more robustly than complex tasks, with pronounced cross-lingual degradation in Tamil and Sinhala that varies by model and problem type. The study argues for fine-grained, type-aware multilingual evaluation to correctly assess cross-lingual mathematical reasoning and to guide future improvements.

Abstract

Large language models (LLMs) demonstrate strong mathematical reasoning in English, but whether these capabilities reflect genuine multilingual reasoning or reliance on translation-based processing in low-resource languages like Sinhala and Tamil remains unclear. We examine this fundamental question by evaluating whether LLMs genuinely reason mathematically in these languages or depend on implicit translation to English-like representations. Using a taxonomy of six math problem types, from basic arithmetic to complex unit conflict and optimization problems, we evaluate four prominent large language models. To avoid translation artifacts that confound language ability with translation quality, we construct a parallel dataset where each problem is natively authored by fluent speakers with mathematical training in all three languages. Our analysis demonstrates that while basic arithmetic reasoning transfers robustly across languages, complex reasoning tasks show significant degradation in Tamil and Sinhala. The pattern of failures varies by model and problem type, suggesting that apparent multilingual competence may not reflect uniform reasoning capabilities across languages. These findings challenge the common assumption that models exhibiting strong multilingual performance can reason equally effectively across languages, and highlight the need for fine-grained, type-aware evaluation in multilingual settings.

Beyond Translation: Evaluating Mathematical Reasoning Capabilities of LLMs in Sinhala and Tamil

TL;DR

This work tackles whether LLMs genuinely reason mathematically in Sinhala and Tamil or rely on translation to English-centric representations. It introduces a six-type taxonomy of math word problems and builds a native-language dataset authored by fluent speakers, enabling zero-shot evaluation of four prominent LLMs. Across languages, basic arithmetic reasoning transfers more robustly than complex tasks, with pronounced cross-lingual degradation in Tamil and Sinhala that varies by model and problem type. The study argues for fine-grained, type-aware multilingual evaluation to correctly assess cross-lingual mathematical reasoning and to guide future improvements.

Abstract

Large language models (LLMs) demonstrate strong mathematical reasoning in English, but whether these capabilities reflect genuine multilingual reasoning or reliance on translation-based processing in low-resource languages like Sinhala and Tamil remains unclear. We examine this fundamental question by evaluating whether LLMs genuinely reason mathematically in these languages or depend on implicit translation to English-like representations. Using a taxonomy of six math problem types, from basic arithmetic to complex unit conflict and optimization problems, we evaluate four prominent large language models. To avoid translation artifacts that confound language ability with translation quality, we construct a parallel dataset where each problem is natively authored by fluent speakers with mathematical training in all three languages. Our analysis demonstrates that while basic arithmetic reasoning transfers robustly across languages, complex reasoning tasks show significant degradation in Tamil and Sinhala. The pattern of failures varies by model and problem type, suggesting that apparent multilingual competence may not reflect uniform reasoning capabilities across languages. These findings challenge the common assumption that models exhibiting strong multilingual performance can reason equally effectively across languages, and highlight the need for fine-grained, type-aware evaluation in multilingual settings.
Paper Structure (22 sections, 3 figures, 1 table)

This paper contains 22 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mathematical Reasoning in LLMs
  4. Multilingual Mathematical Evaluation
  5. Sinhala, and Tamil Evaluation
  6. Methodology
  7. Problem Taxonomy
  8. Type 1: Single-Step Problems.
  9. Type 2: Multi-Step Problems.
  10. Type 3: Problems with Extraneous Information.
  11. Type 4: Unit Conflict Problems.
  12. Type 5: Logical Deduction Problems.
  13. Type 6: Optimization Problems.
  14. Dataset Construction
  15. Evaluation Protocol
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Sample problems from each of the six problem types in English, Sinhala, and Tamil, illustrating the taxonomy's coverage from basic arithmetic (Type 1) to complex optimization (Type 6).
  • Figure 2: Accuracy (%) across four LLMs, six problem types, and three languages. Darker green indicates higher accuracy; red tones highlight degradation.
  • Figure 3: Radar plots comparing model accuracy (%) across six problem types for each language. Polygon shrinkage from English to Sinhala and Tamil reflects cross-lingual performance loss.