MorphPiece : A Linguistic Tokenizer for Large Language Models

TL;DR

This work addresses the limitation of purely statistical tokenizers that ignore morphology by introducing MorphPiece, a tokenizer that combines deterministic morpheme segmentation with BPE. A GPT-2–style model, MorphGPT, trained with MorphPiece, achieves superior or comparable performance to GPT-2 across language modeling, zero-shot GLUE, and extensive sequence-embedding benchmarks, often with far fewer training steps. The results demonstrate that linguistically informed tokenization can improve generalization and efficiency, particularly for morphologically rich text, and suggest broader applicability through detokenization and scalable MorphTable coverage. Overall, MorphPiece motivates incorporating linguistic priors into pretraining to enhance downstream performance and robustness of large language models.

Abstract

Tokenization is a critical part of modern NLP pipelines. However, contemporary tokenizers for Large Language Models are based on statistical analysis of text corpora, without much consideration to the linguistic features. I propose a linguistically motivated tokenization scheme, MorphPiece, which is based partly on morphological segmentation of the underlying text. A GPT-style causal language model trained on this tokenizer (called MorphGPT) shows comparable or superior performance on a variety of supervised and unsupervised NLP tasks, compared to the OpenAI GPT-2 model. Specifically I evaluated MorphGPT on language modeling tasks, zero-shot performance on GLUE Benchmark with various prompt templates, massive text embedding benchmark (MTEB) for supervised and unsupervised performance, and lastly with another morphological tokenization scheme (FLOTA, Hoffmann et al., 2022) and find that the model trained on MorphPiece outperforms GPT-2 on most evaluations, at times with considerable margin, despite being trained for about half the training iterations.

Figures (4)

• Figure 1: MorphPiece tokenization Scheme : After standard BPE pre-tokenization, for each pre-token, we tokenize with MorphTable if the token exists in the MorphTable; if not, we apply standard BPE with custom trained vocabulary
• Figure 2: Example of detokenization. First we mark the types of affixes or 'bpe'. Orange color tokens come from MorphPiece and teal colored come from BPE. Then we follow the arrows from Figure \ref{['fig:detokenizer']} to find word boundaries, which are looked up in reverse-MorphTable to find words.
• Figure 3: Detokenization mechanism from morphemes to English words. Black lines show word continuation; red dashed lines show word boundary, and missing connections imply invalid transitions. Hash denotes compound words. Stem$>$Stem is a special case
• Figure 4: Training performance of GPT-2, GPT-2+FLOTA and MorphGPT on ArXiv-Large datasets.