From BERT to LLMs: Comparing and Understanding Chinese Classifier Prediction in Language Models

TL;DR

This paper investigates how Chinese classifiers are predicted by BERT and autoregressive LLMs, using masking-based probing and attention-manipulation to analyze information contribution and attention dynamics. It shows that BERT consistently outperforms LLMs on classifier prediction, and that bidirectional attention plays a critical role in achieving high accuracy and robust predictions. Fine-tuning improves LLM performance but does not close the gap with BERT, indicating architectural limitations beyond model size. The findings highlight the importance of head-noun dependencies and contextual integration for classifier selection, suggesting future work should enhance bidirectional/contextual capabilities in LLMs for better linguistic competence in Chinese.

Abstract

Classifiers are an important and defining feature of the Chinese language, and their correct prediction is key to numerous educational applications. Yet, whether the most popular Large Language Models (LLMs) possess proper knowledge the Chinese classifiers is an issue that has largely remain unexplored in the Natural Language Processing (NLP) literature. To address such a question, we employ various masking strategies to evaluate the LLMs' intrinsic ability, the contribution of different sentence elements, and the working of the attention mechanisms during prediction. Besides, we explore fine-tuning for LLMs to enhance the classifier performance. Our findings reveal that LLMs perform worse than BERT, even with fine-tuning. The prediction, as expected, greatly benefits from the information about the following noun, which also explains the advantage of models with a bidirectional attention mechanism such as BERT.

Figures (4)

• Figure 1: This figure elaborates on correct syntactic structures of the Chinese classifier system, the types of Chinese classifiers, and corresponding examples, where Num, CL, Dem. Pron., and Approx. stand for numeral, classifier, demonstrative pronoun, and approximation, respectively. Notably, several classifier examples on both sides are accompanied by icons that illustrate the approximate meanings they convey. A detailed explanation is provided in Appendix \ref{['ref: Chinese Classifier Categories and Explanation']}.
• Figure 2: This figure shows the workflow of the project. As denoted by purple arrows, the given sample sentence is input into LMs for prediction. The sentences in the dataset vary in length (and are not necessarily limited to 7 tokens as in the example shown in the figure), while classifiers may consist of either one or two Chinese characters.
• Figure 3: The fine-grained analysis of the six types of classifiers' predictions among the proposed LMs. The black polyline represents the R-rank value (the lower, the better); the bar charts in different colors represent the accuracy of specific models in this type of classifier (the higher, the better). The "ft" in model names represents that the LMs have applied a fine-tuning strategy.
• Figure 4: Line chart of accuracy for used models in groups divided by classifier frequency from high to low. The dashed line represents the fine-tuned model, and the solid line represents the original model.