SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers

TL;DR

The paper addresses the challenge of obtaining robust morphological representations for morphologically rich Uralic languages when clean morpheme lexicons are scarce. It introduces SampoNLP, a corpus-free toolkit that builds high-purity morpheme lexicons via an MDL-inspired Self-Referential Atomicity Scoring pipeline (IMDP), operating entirely on candidate lists without corpus frequencies. Using these lexicons, the authors evaluate BPE tokenizers for Finnish, Hungarian, and Estonian with a unified Integrated Performance Score ($IPS$) that balances Lexical Morpheme Coverage ($LMC$) and Over-Split Rate ($OSR$), identifying language-specific optimal vocabulary ranges ($k^*$). The results show substantial cross-language variation: Hungarian attains the strongest performance (max $IPS \approx 0.73$) while Finnish and Estonian lag (max $IPS \approx 0.31$ and $0.39$ respectively), underscoring limitations of standard BPE for highly agglutinative systems. The work provides practical, data-driven guidance for tokenizer vocabulary sizing and delivers open-source tools and resources to promote reproducible, morphologically aware tokenization research in low-resource and data-scarce settings.

Abstract

The quality of subword tokenization is critical for Large Language Models, yet evaluating tokenizers for morphologically rich Uralic languages is hampered by the lack of clean morpheme lexicons. We introduce SampoNLP, a corpus-free toolkit for morphological lexicon creation using MDL-inspired Self-Referential Atomicity Scoring, which filters composite forms through internal structural cues - suited for low-resource settings. Using the high-purity lexicons generated by SampoNLP for Finnish, Hungarian, and Estonian, we conduct a systematic evaluation of BPE tokenizers across a range of vocabulary sizes (8k-256k). We propose a unified metric, the Integrated Performance Score (IPS), to navigate the trade-off between morpheme coverage and over-splitting. By analyzing the IPS curves, we identify the "elbow points" of diminishing returns and provide the first empirically grounded recommendations for optimal vocabulary sizes (k) in these languages. Our study not only offers practical guidance but also quantitatively demonstrates the limitations of standard BPE for highly agglutinative languages. The SampoNLP library and all generated resources are made publicly available: https://github.com/AragonerUA/SampoNLP

Figures (6)

• Figure 1: An overview of the Iterative Morphological Decomposition Pipeline (IMDP).
• Figure 2: Lexical Morpheme Coverage (LMC) across different vocabulary sizes (k). LMC represents the percentage of reference morphemes found as single, complete tokens in the tokenizer's vocabulary.
• Figure 3: Over-Split Rate (OSR) as a function of vocabulary size (k). OSR denotes the fraction of reference morphemes that occur in words but never appear as a single token in any tokenization.
• Figure 4: IPS vs. vocabulary size (k) for Hungarian. Hungarian shows the most consistent improvement in IPS, reflecting its comparatively transparent agglutinative structure with fewer morphophonological alternations. The elbow point is at 80k, and the 90% quality threshold at 128k, yielding a recommended range of 80k–128k.
• Figure 5: IPS vs. vocabulary size (k) for Estonian. While the overall pattern of diminishing returns is similar to Hungarian, the lower IPS plateau indicates reduced learnability due to Estonian’s extensive morphophonological alternations, which obscure orthographic morpheme boundaries. The recommended range remains 80k–128k.