From Smør-re-brød to Subwords: Training LLMs on Danish, One Morpheme at a Time

TL;DR

This work investigates morphology-informed tokenization for Danish by developing Morfessor-guided tokenizers and evaluating them on downstream generative models. Using a semi-supervised Morfessor setup and a small annotated morpheme dataset, the authors show substantial segmentation gains (F1 up to $58.84$) compared to Danish BPE baselines ($39.28$). Across two transformer models, Morph-based tokenizers improve linguistic acceptability and human evaluation scores, and full-model fine-tuning appears to be crucial for realizing these gains. The findings support incorporating Danish morphological segmentation into tokenization to enhance generative Danish language modeling, with implications for low-resource languages and linguistically informed NLP tooling.

Abstract

The best performing transformer-based language models use subword tokenization techniques, such as Byte-Pair-Encoding (BPE). However, these approaches often overlook linguistic principles, such as morphological segmentation, which we believe is fundamental for understanding language-specific word structure. In this study, we leverage an annotated Danish morphological dataset to train a semisupervised model for morphological segmentation, enabling the development of tokenizers optimized for Danish morphology. We evaluate four distinct tokenizers, including two custom morphological tokenizers, by analyzing their performance in morphologically segmenting Danish words. Additionally, we train two generative transformer models, \textit{CerebrasGPT-111M} and \textit{LLaMA-3.2 1B}, using these tokenizers and evaluate their downstream performance. Our findings reveal that our custom-developed tokenizers substantially enhance morphological segmentation, achieving an F1 score of 58.84, compared to 39.28 achieved by a Danish BPE tokenizer. In downstream tasks, models trained with our morphological tokenizers outperform those using BPE tokenizers across different evaluation metrics. These results highlight that incorporating Danish morphological segmentation strategies into tokenizers leads to improved performance in generative transformer models on Danish language

Figures (2)

• Figure 1: Morphological PreTrainedTokenizer implementation. The tokenizer prioritizes morpheme segmentation, then defaulting to the Morfessor model. The Mixed tokenizer uses BPE as a fallback if the Morfessor split is not found in the vocabulary
• Figure 2: F1 score for each Morfessor model and their amount of annotated data. F1 counts the correct segments.