Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs for Translation: Historical, Low-Resourced Languages and Contemporary AI Models

Merve Tekgurler

TL;DR

This work examines the translation of an 18th century Ottoman Turkish memoir using Google Gemini 1.5 Pro to English and analyzes how AI safety content moderation impacts translation of historical and context rich material. Through a two pass translation protocol on OT EN and parallel DE EN text pairs created via SentAlign and LaBSE, the study reveals that safety flags rendered a nontrivial fraction of sentences untranslated and that severity relates to model confidence in safety judgments. The findings show similar safety flag patterns across OT and DE, indicating that content related to violence and sexual content drives blocking more than language resource limitations. The work highlights the need to balance historical translation fidelity with robust safety controls and points to the importance of designing translations pipelines that preserve victims voices in war and other sensitive contexts.

Abstract

Large Language Models (LLMs) have demonstrated remarkable adaptability in performing various tasks, including machine translation (MT), without explicit training. Models such as OpenAI's GPT-4 and Google's Gemini are frequently evaluated on translation benchmarks and utilized as translation tools due to their high performance. This paper examines Gemini's performance in translating an 18th-century Ottoman Turkish manuscript, Prisoner of the Infidels: The Memoirs of Osman Agha of Timisoara, into English. The manuscript recounts the experiences of Osman Agha, an Ottoman subject who spent 11 years as a prisoner of war in Austria, and includes his accounts of warfare and violence. Our analysis reveals that Gemini's safety mechanisms flagged between 14 and 23 percent of the manuscript as harmful, resulting in untranslated passages. These safety settings, while effective in mitigating potential harm, hinder the model's ability to provide complete and accurate translations of historical texts. Through real historical examples, this study highlights the inherent challenges and limitations of current LLM safety implementations in the handling of sensitive and context-rich materials. These real-world instances underscore potential failures of LLMs in contemporary translation scenarios, where accurate and comprehensive translations are crucial-for example, translating the accounts of modern victims of war for legal proceedings or humanitarian documentation.

LLMs for Translation: Historical, Low-Resourced Languages and Contemporary AI Models

TL;DR

This work examines the translation of an 18th century Ottoman Turkish memoir using Google Gemini 1.5 Pro to English and analyzes how AI safety content moderation impacts translation of historical and context rich material. Through a two pass translation protocol on OT EN and parallel DE EN text pairs created via SentAlign and LaBSE, the study reveals that safety flags rendered a nontrivial fraction of sentences untranslated and that severity relates to model confidence in safety judgments. The findings show similar safety flag patterns across OT and DE, indicating that content related to violence and sexual content drives blocking more than language resource limitations. The work highlights the need to balance historical translation fidelity with robust safety controls and points to the importance of designing translations pipelines that preserve victims voices in war and other sensitive contexts.

Abstract

Large Language Models (LLMs) have demonstrated remarkable adaptability in performing various tasks, including machine translation (MT), without explicit training. Models such as OpenAI's GPT-4 and Google's Gemini are frequently evaluated on translation benchmarks and utilized as translation tools due to their high performance. This paper examines Gemini's performance in translating an 18th-century Ottoman Turkish manuscript, Prisoner of the Infidels: The Memoirs of Osman Agha of Timisoara, into English. The manuscript recounts the experiences of Osman Agha, an Ottoman subject who spent 11 years as a prisoner of war in Austria, and includes his accounts of warfare and violence. Our analysis reveals that Gemini's safety mechanisms flagged between 14 and 23 percent of the manuscript as harmful, resulting in untranslated passages. These safety settings, while effective in mitigating potential harm, hinder the model's ability to provide complete and accurate translations of historical texts. Through real historical examples, this study highlights the inherent challenges and limitations of current LLM safety implementations in the handling of sensitive and context-rich materials. These real-world instances underscore potential failures of LLMs in contemporary translation scenarios, where accurate and comprehensive translations are crucial-for example, translating the accounts of modern victims of war for legal proceedings or humanitarian documentation.

Paper Structure

This paper contains 17 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Translation with LLMs
  4. Ottoman Turkish
  5. AI Safety and Content Moderation
  6. Data
  7. Osman Agha: Person and Manuscript
  8. Dataset
  9. Preliminary Experiments
  10. Methods
  11. Results
  12. Analysis
  13. Discussion
  14. Implications
  15. Ethical Considerations
  16. ...and 2 more sections

Figures (6)

  • Figure 1: Severity Score - Probability Score Plot for Flagged Ottoman Turkish Sentences
  • Figure 2: Severity Score - Probability Score Plot for Flagged German Sentences
  • Figure 3: Comparison of the Severity Scores in Ottoman and German Datasets
  • Figure 4: Distribution of Flagged Sentences across the entire Ottoman Manuscript
  • Figure 5: Distribution of Flagged Sentences across the entire German Manuscript
  • ...and 1 more figures