Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Language Identification for Romansh Varieties

Charlotte Model, Sina Ahmadi, Jannis Vamvas

Abstract

The Romansh language has several regional varieties, called idioms, which sometimes have limited mutual intelligibility. Despite this linguistic diversity, there has been a lack of documented efforts to build a language identification (LID) system that can distinguish between these idioms. Since Romansh LID should also be able to recognize Rumantsch Grischun, a supra-regional variety that combines elements of several idioms, this makes for a novel and interesting classification problem. In this paper, we present a LID system for Romansh idioms based on an SVM approach. We evaluate our model on a newly curated benchmark across two domains and find that it reaches an average in-domain accuracy of 97%, enabling applications such as idiom-aware spell checking or machine translation. Our classifier is publicly available.

Robust Language Identification for Romansh Varieties

Abstract

The Romansh language has several regional varieties, called idioms, which sometimes have limited mutual intelligibility. Despite this linguistic diversity, there has been a lack of documented efforts to build a language identification (LID) system that can distinguish between these idioms. Since Romansh LID should also be able to recognize Rumantsch Grischun, a supra-regional variety that combines elements of several idioms, this makes for a novel and interesting classification problem. In this paper, we present a LID system for Romansh idioms based on an SVM approach. We evaluate our model on a newly curated benchmark across two domains and find that it reaches an average in-domain accuracy of 97%, enabling applications such as idiom-aware spell checking or machine translation. Our classifier is publicly available.
Paper Structure (20 sections, 1 figure, 6 tables)

This paper contains 20 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Romansh and its Varieties
  3. Related Work
  4. Language Identification
  5. Romansh NLP
  6. Data
  7. Preprocessing
  8. Named-Entity Masking
  9. Experimental Setup
  10. Data Splits
  11. Classification
  12. Hyperparameter Optimization
  13. Results
  14. Overall Results
  15. Per Idiom Performances
  16. ...and 5 more sections

Figures (1)

  • Figure 1: Row-normalized confusion matrices for all test sets. The model achieves near-perfect classification on the balanced in-domain set (test-b), while confusion increases on out-of-domain data (test-d).