Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MITRA: A Large-Scale Parallel Corpus and Multilingual Pretrained Language Model for Machine Translation and Semantic Retrieval for Pāli, Sanskrit, Buddhist Chinese, and Tibetan

Sebastian Nehrdich, Kurt Keutzer

TL;DR

MITRA addresses the scarcity of multilingual parallel data in ancient Buddhist texts by mining a large-scale parallel corpus and training a domain-specific LLM. The pipeline pivots MT to English to identify cross-language candidates, refines them with sentence alignment, and yields 1.74M parallel sentence pairs. Gemma 2 MITRA is trained as a 9B-parameter base model and fine-tuned for machine translation and semantic retrieval, achieving state-of-the-art open-model MT and superior retrieval across Sanskrit, Pāli, Tibetan, and Buddhist Chinese. The authors provide open access to the dataset, model weights, and benchmarks, enabling replication and broad philological and NLP research, while acknowledging limitations and proposing future expansions to additional languages and model distillation.

Abstract

Ancient Buddhist literature features frequent, yet often unannotated, textual parallels spread across diverse languages: Sanskrit, Pāli, Buddhist Chinese, Tibetan, and more. The scale of this material makes manual examination prohibitive. We present the MITRA framework, which consists of a novel pipeline for multilingual parallel passage mining, MITRA-parallel, a large-scale corpus of 1.74 million parallel sentence pairs between Sanskrit, Chinese, and Tibetan, and the development of the domain-specific pretrained language model Gemma 2 MITRA. We present Gemma 2 MITRA-MT, a version of this base model fine-tuned on machine translation tasks, reaching state-of-the-art performance for machine translation of these languages into English and outperforming even much larger open-source models. We also present Gemma 2 MITRA-E, a semantic embedding model that shows state-of-the-art performance on a novel, detailed semantic embedding benchmark. We make the parallel dataset, model weights, and semantic similarity benchmark openly available to aid both NLP research and philological studies in Buddhist and classical Asian literature.

MITRA: A Large-Scale Parallel Corpus and Multilingual Pretrained Language Model for Machine Translation and Semantic Retrieval for Pāli, Sanskrit, Buddhist Chinese, and Tibetan

TL;DR

MITRA addresses the scarcity of multilingual parallel data in ancient Buddhist texts by mining a large-scale parallel corpus and training a domain-specific LLM. The pipeline pivots MT to English to identify cross-language candidates, refines them with sentence alignment, and yields 1.74M parallel sentence pairs. Gemma 2 MITRA is trained as a 9B-parameter base model and fine-tuned for machine translation and semantic retrieval, achieving state-of-the-art open-model MT and superior retrieval across Sanskrit, Pāli, Tibetan, and Buddhist Chinese. The authors provide open access to the dataset, model weights, and benchmarks, enabling replication and broad philological and NLP research, while acknowledging limitations and proposing future expansions to additional languages and model distillation.

Abstract

Ancient Buddhist literature features frequent, yet often unannotated, textual parallels spread across diverse languages: Sanskrit, Pāli, Buddhist Chinese, Tibetan, and more. The scale of this material makes manual examination prohibitive. We present the MITRA framework, which consists of a novel pipeline for multilingual parallel passage mining, MITRA-parallel, a large-scale corpus of 1.74 million parallel sentence pairs between Sanskrit, Chinese, and Tibetan, and the development of the domain-specific pretrained language model Gemma 2 MITRA. We present Gemma 2 MITRA-MT, a version of this base model fine-tuned on machine translation tasks, reaching state-of-the-art performance for machine translation of these languages into English and outperforming even much larger open-source models. We also present Gemma 2 MITRA-E, a semantic embedding model that shows state-of-the-art performance on a novel, detailed semantic embedding benchmark. We make the parallel dataset, model weights, and semantic similarity benchmark openly available to aid both NLP research and philological studies in Buddhist and classical Asian literature.
Paper Structure (25 sections, 2 figures, 5 tables)

This paper contains 25 sections, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Previous Research
  3. MITRA-parallel dataset
  4. Data Mining Procedure
  5. Machine Translation
  6. Candidate Clusters
  7. Sentence Alignment
  8. Generated Dataset
  9. Gemma 2 MITRABase LLM
  10. Monolingual data
  11. Parallel data
  12. Continuous Pretraining
  13. Machine Translation Instruction Fine-tuning
  14. Semantic Retrieval Fine-tuning
  15. Machine Translation Evaluation
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Number of datapoints per language pair in the sentence-level aligned parallel dataset for Ancient Buddhist Languages.
  • Figure 2: Machine translation performance of different open base models compared to our finetuned model. Performance is measure in GEMBA score, which we implemented with Gemini 2.0 Flash as judge.