Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing and Improving Punctuation Robustness in English-Marathi Machine Translation

Kaustubh Shivshankar Shejole, Sourabh Deoghare, Pushpak Bhattacharyya

TL;DR

This work tackles punctuation-driven ambiguity in English–Marathi MT by introducing Virām, a diagnostic benchmark of 54 punctuation-ambiguous instances. It compares two strategies—a restore-then-translate pipeline and direct fine-tuning on mixed punctuation data—and analyzes performance on Virām and standard benchmarks. The findings show significant gains from punctuation-aware models and pipelines, while current LLMs underperform compared with task-specific approaches in preserving meaning under punctuation variability. The study highlights practical implications for punctuation robustness in Indic MT and points to future directions in evaluation, hybrid architectures, and cross-language application.

Abstract

Punctuation plays a critical role in resolving semantic and structural ambiguity in written language. Machine Translation (MT) systems are now widely applied across diverse domains and languages, including many low-resource settings. In this work, we focus on Marathi, a low- to middle-resource language. We introduce Virām, the first diagnostic benchmark for assessing punctuation robustness in English-to-Marathi machine translation, consisting of 54 manually curated, punctuation-ambiguous instances. We evaluate two primary strategies for enhancing reliability: a pipeline-based restore-then-translate approach and direct fine-tuned on punctuation-varied data. Our results demonstrate that specialized fine-tuned models and pipeline systems significantly improve translation quality over standard baselines on the Virām benchmark. Qualitative analysis reveals that the original model may result in wrong translations leading to wrong interpretations, while fine-tuned models significantly improve overall reliability. Furthermore, we find that current Large Language Models (LLMs) lag behind these task-specific approaches in preserving meaning for punctuation-ambiguous text, thus necessitating further research in this area.

Assessing and Improving Punctuation Robustness in English-Marathi Machine Translation

TL;DR

This work tackles punctuation-driven ambiguity in English–Marathi MT by introducing Virām, a diagnostic benchmark of 54 punctuation-ambiguous instances. It compares two strategies—a restore-then-translate pipeline and direct fine-tuning on mixed punctuation data—and analyzes performance on Virām and standard benchmarks. The findings show significant gains from punctuation-aware models and pipelines, while current LLMs underperform compared with task-specific approaches in preserving meaning under punctuation variability. The study highlights practical implications for punctuation robustness in Indic MT and points to future directions in evaluation, hybrid architectures, and cross-language application.

Abstract

Punctuation plays a critical role in resolving semantic and structural ambiguity in written language. Machine Translation (MT) systems are now widely applied across diverse domains and languages, including many low-resource settings. In this work, we focus on Marathi, a low- to middle-resource language. We introduce Virām, the first diagnostic benchmark for assessing punctuation robustness in English-to-Marathi machine translation, consisting of 54 manually curated, punctuation-ambiguous instances. We evaluate two primary strategies for enhancing reliability: a pipeline-based restore-then-translate approach and direct fine-tuned on punctuation-varied data. Our results demonstrate that specialized fine-tuned models and pipeline systems significantly improve translation quality over standard baselines on the Virām benchmark. Qualitative analysis reveals that the original model may result in wrong translations leading to wrong interpretations, while fine-tuned models significantly improve overall reliability. Furthermore, we find that current Large Language Models (LLMs) lag behind these task-specific approaches in preserving meaning for punctuation-ambiguous text, thus necessitating further research in this area.
Paper Structure (28 sections, 8 figures, 8 tables)

This paper contains 28 sections, 8 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Creating the Virām Benchmark
  4. Methodology
  5. Approach 1: Restore Punctuation then Translate
  6. Approach 2: Direct Translation
  7. Prompting LLMs
  8. Results and Analysis
  9. Quantitative Performance
  10. Analysis of Prompting Strategies in LLMs
  11. Performance Analysis on Standard Benchmarks
  12. Qualitative Analysis
  13. Conclusion and Future Directions
  14. More details about the Virām Benchmark
  15. Annotation Procedure
  16. ...and 13 more sections

Figures (8)

  • Figure 1: A missing comma can lead to a disaster in English–Marathi machine translation.
  • Figure 2: Data handling for the punctuation restoration task: Approach 1.
  • Figure 3: Data handling for the direct fine-tuning task: Approach 2.
  • Figure 4: Prompt used for original translation.
  • Figure 5: Prompt used for zero-shot reasoning with Approach 1 (Restore then Translate)
  • ...and 3 more figures