Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Subword Embedding Approach for Variation Detection in Luxembourgish User Comments

Anne-Marie Lutgen, Alistair Plum, Christoph Purschke

TL;DR

This work reframes orthographic and lexical variation in Luxembourgish as a signal rather than noise and introduces a reproducible, unsupervised pipeline that discovers variant families directly from raw text. It combines subword embeddings with a joint cosine and n-gram similarity framework to cluster related forms, followed by a constrained pruning and a qualitative analysis to interpret the results. Applied to a large RTL comment corpus, the approach yields about 800 variant families across multiple linguistic facets (orthographic, morphological, lexical, collocation, tokenisation, regional, and other), revealing systematic patterns aligned with sociolinguistic expectations. The method is particularly suited to non-standardised, high-variation domains and under-resourced varieties, offering a transferable framework for studying language variety in multilingual or small-language contexts, with results supported by transparent scoring and publicly released code.

Abstract

This paper presents an embedding-based approach to detecting variation without relying on prior normalisation or predefined variant lists. The method trains subword embeddings on raw text and groups related forms through combined cosine and n-gram similarity. This allows spelling and morphological diversity to be examined and analysed as linguistic structure rather than treated as noise. Using a large corpus of Luxembourgish user comments, the approach uncovers extensive lexical and orthographic variation that aligns with patterns described in dialectal and sociolinguistic research. The induced families capture systematic correspondences and highlight areas of regional and stylistic differentiation. The procedure does not strictly require manual annotation, but does produce transparent clusters that support both quantitative and qualitative analysis. The results demonstrate that distributional modelling can reveal meaningful patterns of variation even in ''noisy'' or low-resource settings, offering a reproducible methodological framework for studying language variety in multilingual and small-language contexts.

A Subword Embedding Approach for Variation Detection in Luxembourgish User Comments

TL;DR

This work reframes orthographic and lexical variation in Luxembourgish as a signal rather than noise and introduces a reproducible, unsupervised pipeline that discovers variant families directly from raw text. It combines subword embeddings with a joint cosine and n-gram similarity framework to cluster related forms, followed by a constrained pruning and a qualitative analysis to interpret the results. Applied to a large RTL comment corpus, the approach yields about 800 variant families across multiple linguistic facets (orthographic, morphological, lexical, collocation, tokenisation, regional, and other), revealing systematic patterns aligned with sociolinguistic expectations. The method is particularly suited to non-standardised, high-variation domains and under-resourced varieties, offering a transferable framework for studying language variety in multilingual or small-language contexts, with results supported by transparent scoring and publicly released code.

Abstract

This paper presents an embedding-based approach to detecting variation without relying on prior normalisation or predefined variant lists. The method trains subword embeddings on raw text and groups related forms through combined cosine and n-gram similarity. This allows spelling and morphological diversity to be examined and analysed as linguistic structure rather than treated as noise. Using a large corpus of Luxembourgish user comments, the approach uncovers extensive lexical and orthographic variation that aligns with patterns described in dialectal and sociolinguistic research. The induced families capture systematic correspondences and highlight areas of regional and stylistic differentiation. The procedure does not strictly require manual annotation, but does produce transparent clusters that support both quantitative and qualitative analysis. The results demonstrate that distributional modelling can reveal meaningful patterns of variation even in ''noisy'' or low-resource settings, offering a reproducible methodological framework for studying language variety in multilingual and small-language contexts.
Paper Structure (25 sections, 2 equations, 2 figures, 4 tables)

This paper contains 25 sections, 2 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Distributional Embeddings
  5. Stage 1: Training Subword Embeddings
  6. Stage 2: Identifying Variant Families
  7. Scoring and Pruning
  8. Dimension-Based Aggregation
  9. Configuration and Output
  10. Stage 3: Qualitative Analysis
  11. Luxembourgish User Comments
  12. Orthographic
  13. Morphological
  14. Lexical
  15. Collocation
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Map of muer variants Gilles2021
  • Figure 2: Map of schnell variants Gilles2021