A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese

TL;DR

This work introduces ZhoBLiMP, the largest Chinese linguistic minimal-pair benchmark, and a family of Zh-Pythia LMs trained from scratch to study Chinese syntax acquisition. It identifies length-related biases in minimal-pair evaluations and proposes SLLN-LP, a sublinear length normalization function, to debias assessments across tokenizers and scales. The results show that many Chinese syntactic phenomena are learnable by relatively small LMs, but Anaphor, Ellipsis, and Quantifiers remain challenging even for large models, highlighting the role of discourse information. The study also provides methodological guidance on linking functions and cross-language benchmarking, along with publicly available data and code to support future research in multilingual minimal-pair evaluation.

Abstract

We present ZhoBLiMP, the largest linguistic minimal pair benchmark for Chinese, with over 100 paradigms, ranging from topicalization to the \textit{Ba} construction. We then train from scratch a suite of Chinese language models (LMs) with different tokenizers, parameter sizes, and token volumes, to study the learning curves of LMs on Chinese. To mitigate the biases introduced by unequal lengths of the sentences in a minimal pair, we propose a new metric named sub-linear length normalized log-probabilities (SLLN-LP). Using SLLN-LP as the metric, our results show that \textsc{Anaphor}, \textsc{Quantifiers}, and \textsc{Ellipsis} in Chinese are difficult for LMs even up to 32B parameters, and that SLLN-LP successfully mitigates biases in ZhoBLiMP, JBLiMP and BLiMP. We conclude that future evaluations should be more carefully designed to consider the intricate relations between linking functions, LMs, and targeted minimal pairs.

Figures (4)

• Figure 1: Data generation procedure illustration. Without writing any codes, linguists can easily generate sentence pairs by crafting grammar templates and vocabulary in the green blocks.
• Figure 2: Effectiveness of debiasing length normalization using different $\alpha$. We report two metrics: (1) average accuracy on $\mathcal{D}_{+}$ and $\mathcal{D}_{-}$ (when minimal pairs have unequal lengths), and (2) $\Delta_{acc}$. left is the average results of 20 Zh-Pythia LMs with the shaded area denoting the standard deviation; right is the results of mpp benchmarks in other languages, including JBLiMP, BLiMP, and BLiMP-NL.
• Figure 3: Phenomenon-specific accuracy on ZhoBLiMP plotted against training FLOPs (log scale). Each point represents a distinct Zh-Pythia LM, with models of identical parameter sizes shown in the same color. Points connected by dotted lines represent models of the same size, where higher training FLOPs indicate larger volumes of training tokens. We also plot the performance of Qwen2.5-14B and human as references.
• Figure 4: Learning curves across training tokens (calculated from training steps). We analyze 47 intermediate checkpoints from Zh-Pythia models trained on 3B tokens. The plot shows average accuracy across 15 LMs (5 model sizes × 3 seeds) with interpolation smoothing. The shaded area represents the standard deviation among checkpoints at equivalent training volumes.