Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features

TL;DR

The paper proposes a novel metric to evaluate the morphological plausibility of subword tokenization without requiring gold morpheme segmentation. It aligns subword tokens to morpho-syntactic features using IBM Model 1 and aggregates the resulting probabilities into a single alignment score, enabling cross-lingual assessment using UniMorph features. Empirical results show the metric correlates strongly with traditional boundary recall across languages with varied morphology, particularly when using Split feature representations and certain aggregation functions. This approach expands cross-lingual evaluation capabilities for subword tokenizers and provides a practical tool for comparing tokenization schemes across diverse languages, with data and code released for replication.

Abstract

We present a novel metric for the evaluation of the morphological plausibility of subword segmentation. Unlike the typically used morpheme boundary or retrieval F-score, which requires gold segmentation data that is either unavailable or of inconsistent quality across many languages, our approach utilizes morpho-syntactic features. These are available in resources such as Universal Dependencies or UniMorph for a much wider range of languages. The metric works by probabilistically aligning subwords with morphological features through an IBM Model 1. Our experiments show that the metric correlates well with traditional morpheme boundary recall while being more broadly applicable across languages with different morphological systems.

Figures (6)

• Figure 1: The workflow for subword tokenization evaluation for morphological plausibility, illustrated with an example of the German verb HTML]d5e8d4mieten ("to rent"): Instead of comparing subwords with morpheme segmentation, we align morpho-syntactic features with subwords using IBM Model 1 to compute our proposed HTML]FF9933alignment score.
• Figure 2: The Spearman correlation of our metric scores with boundary recall, precision, and F$_1$ score for Finnish. Plots for other languages are in the Appendix.
• Figure 3: The correlation of the alignment-based score with boundary recall for all languages.
• Figure 4: The correlation of the alignment-based score with boundary precision for all languages.
• Figure 5: The correlation of the alignment-based score with F$_1$ score for all languages.