The Effect of Data Partitioning Strategy on Model Generalizability: A Case Study of Morphological Segmentation

TL;DR

This paper investigates how data partitioning strategies affect generalizability in cross-linguistic morphological segmentation, focusing on random versus adversarial splits across 19 languages and four architectures. Using a large, typologically diverse dataset and a robust experimental design, it shows that random splits yield higher and more stable $F1$ scores on new data and produce more consistent model rankings than adversarial splits. The study provides strong empirical evidence that partitioning strategy materially influences evaluation reliability, with implications for cross-language NLP benchmarking and language-tech development for indigenous languages. Limitations include same-domain data constraints and no parameter tuning, suggesting future work should extend to additional tasks and domain shifts to validate generalizability further.

Abstract

Recent work to enhance data partitioning strategies for more realistic model evaluation face challenges in providing a clear optimal choice. This study addresses these challenges, focusing on morphological segmentation and synthesizing limitations related to language diversity, adoption of multiple datasets and splits, and detailed model comparisons. Our study leverages data from 19 languages, including ten indigenous or endangered languages across 10 language families with diverse morphological systems (polysynthetic, fusional, and agglutinative) and different degrees of data availability. We conduct large-scale experimentation with varying sized combinations of training and evaluation sets as well as new test data. Our results show that, when faced with new test data: (1) models trained from random splits are able to achieve higher numerical scores; (2) model rankings derived from random splits tend to generalize more consistently.

Figures (2)

• Figure 1: Simple illustration of a single dataset construction process for our experiments: given the original dataset of a language, a new test sample is constructed either randomly or adversarially (in this case, the new test sample accounts for 50% of the original dataset); the residual data from the original data is then divided into a training and an evaluation set (E) at a fixed 9:1 ratio, via random or adversarial splits.
• Figure 2: Score variability for every eval set size (%) given each data partitioning strategy averaged across languages and model architectures, when the new test samples are generated randomly.