Subword-Based Comparative Linguistics across 242 Languages Using Wikipedia Glottosets

TL;DR

The paper presents a scalable, script-driven framework for cross-linguistic comparison across 242 languages by building monolingual glottosets from Wikipedia and applying BPE to derive rank-based subword representations. It demonstrates that BPE boundaries align with morphologically meaningful units and that BPE vocabulary similarity exhibits a significant though moderate phylogenetic signal (Mantel $r = 0.329$, $p < 0.001$), while revealing substantial cross-language segmentation variation on homographs. The approach yields practical benefits for low-resource language technology, including a $44\times$ improvement in unsupervised language identification across $321$ Latin-script languages and robust discrimination of cross-linguistic forms. Overall, the work provides macro-level lexical insights and a unified, data-driven framework bridging phylogenetic and typological perspectives, with clear pathways for extending to broader web data and typology databases.

Abstract

We present a large-scale comparative study of 242 Latin and Cyrillic-script languages using subword-based methodologies. By constructing 'glottosets' from Wikipedia lexicons, we introduce a framework for simultaneous cross-linguistic comparison via Byte-Pair Encoding (BPE). Our approach utilizes rank-based subword vectors to analyze vocabulary overlap, lexical divergence, and language similarity at scale. Evaluations demonstrate that BPE segmentation aligns with morpheme boundaries 95% better than random baseline across 15 languages (F1 = 0.34 vs 0.15). BPE vocabulary similarity correlates significantly with genetic language relatedness (Mantel r = 0.329, p < 0.001), with Romance languages forming the tightest cluster (mean distance 0.51) and cross-family pairs showing clear separation (0.82). Analysis of 26,939 cross-linguistic homographs reveals that 48.7% receive different segmentations across related languages, with variation correlating to phylogenetic distance. Our results provide quantitative macro-linguistic insights into lexical patterns across typologically diverse languages within a unified analytical framework.

Figures (4)

• Figure 1: Pipeline architecture for subword-based comparative linguistics across 242 languages using Wikipedia glottosets. The workflow illustrates the transformation of Wikipedia dumps (320 languages) through sequential stages: script-based filtering yielding 37 Cyrillic and 205 Latin script languages, monolingual glottoset construction with TF/DF metrics, BPE tokenization (both individual and combined training with 4096 tokens vocabulary), and vector-based subword analysis. Each colored node represents a distinct processing phase, culminating in macro-level insights for script-level comparative linguistics. This modular approach enables scalable analysis of morphological patterns across multiple languages simultaneously.
• Figure 2: Visualization of interactive BPE merge graphs for Ukrainian (left) and Finnish (right) subword tokenization patterns. The diagrams show directed merge sequences with reuse counts indicated in parentheses. Vertical black bars represent merge steps, while edges show the progression of subword unit formation. The contrasting patterns reflect language-specific morphological characteristics: Ukrainian showing consistent Cyrillic character combinations, while Finnish exhibits agglutinative patterns. Interactive web application available on our repository.
• Figure 3: Hierarchical BPE tokenization trees comparing the word T2A заказала in Ukrainian (left) and Russian (right). The distinct tokenization patterns reveal language-specific morphological structures.
• Figure 4: Hierarchical BPE tokenization trees for the word T2A "промисловість" (industry) across three East Slavic languages. The Ukrainian tokenization produces semantically consistent morphemes, while Belarusian and Russian models generate more fragmented subword units.