Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Mediomatix Corpus: Parallel Data for Romansh Language Varieties via Comparable Schoolbooks

Zachary Hopton, Jannis Vamvas, Andrin Büchler, Anna Rutkiewicz, Rico Cathomas, Rico Sennrich

TL;DR

The paper introduces Mediomatix, the first large-scale parallel corpus for all five Romansh idioms, derived from 291 schoolbooks and totaling over 2 million tokens across 207k multi-parallel segments. It develops a robust alignment pipeline based on VecAlign with pivot-consensus alignment and a length-filtering heuristic, achieving high precision validated by automatic metrics and human evaluation. The resource is demonstrated for machine translation across idioms using both a fine-tuned NLLB model and GPT-4o family variants, showing notable improvements and establishing a baseline for cross-idiom Romansh NLP. This dataset enables MT, data augmentation, and broader NLP applications for Romansh idioms, while acknowledging limitations such as translationese, licensing, and content coverage, with ongoing release and expansion planned.

Abstract

The five idioms (i.e., varieties) of the Romansh language are largely standardized and are taught in the schools of the respective communities in Switzerland. In this paper, we present the first parallel corpus of Romansh idioms. The corpus is based on 291 schoolbook volumes, which are comparable in content for the five idioms. We use automatic alignment methods to extract 207k multi-parallel segments from the books, with more than 2M tokens in total. A small-scale human evaluation confirms that the segments are highly parallel, making the dataset suitable for NLP applications such as machine translation between Romansh idioms. We release the parallel and unaligned versions of the dataset under a CC-BY-NC-SA license and demonstrate its utility for machine translation by training and evaluating an LLM and a supervised multilingual MT model on the dataset.

The Mediomatix Corpus: Parallel Data for Romansh Language Varieties via Comparable Schoolbooks

TL;DR

The paper introduces Mediomatix, the first large-scale parallel corpus for all five Romansh idioms, derived from 291 schoolbooks and totaling over 2 million tokens across 207k multi-parallel segments. It develops a robust alignment pipeline based on VecAlign with pivot-consensus alignment and a length-filtering heuristic, achieving high precision validated by automatic metrics and human evaluation. The resource is demonstrated for machine translation across idioms using both a fine-tuned NLLB model and GPT-4o family variants, showing notable improvements and establishing a baseline for cross-idiom Romansh NLP. This dataset enables MT, data augmentation, and broader NLP applications for Romansh idioms, while acknowledging limitations such as translationese, licensing, and content coverage, with ongoing release and expansion planned.

Abstract

The five idioms (i.e., varieties) of the Romansh language are largely standardized and are taught in the schools of the respective communities in Switzerland. In this paper, we present the first parallel corpus of Romansh idioms. The corpus is based on 291 schoolbook volumes, which are comparable in content for the five idioms. We use automatic alignment methods to extract 207k multi-parallel segments from the books, with more than 2M tokens in total. A small-scale human evaluation confirms that the segments are highly parallel, making the dataset suitable for NLP applications such as machine translation between Romansh idioms. We release the parallel and unaligned versions of the dataset under a CC-BY-NC-SA license and demonstrate its utility for machine translation by training and evaluating an LLM and a supervised multilingual MT model on the dataset.

Paper Structure

This paper contains 38 sections, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Language Situation
  4. Romansh Idioms
  5. Contemporary Role of Romansh Idioms
  6. NLP for Romansh
  7. Teaching Materials as a Source of Parallel Text for Romansh Idioms
  8. Dataset Creation
  9. Extraction of Text Segments
  10. Multi-parallel Alignment
  11. Bilingual Alignments
  12. Choice of Embedding Model
  13. Pivot Consensus Alignment
  14. Length Heuristic
  15. Validation of Alignment Quality
  16. ...and 23 more sections

Figures (2)

  • Figure 1: Example of a parallel segment in the five Romansh idioms.
  • Figure 2: Depiction of multi-parallel alignment via a pivot idiom, as described in Section \ref{['sec:pivot']}. The four two-column tables represent bilingual alignments with a given pivot idiom (highlighted in orange). The five-column table shows the result of the full outer join of the bilingual alignments on the pivot idiom, which we take as the multi-parallel alignment for this pivot idiom.