CPG-EVAL: A Multi-Tiered Benchmark for Evaluating the Chinese Pedagogical Grammar Competence of Large Language Models
Dong Wang
TL;DR
CPG-EVALAddress the lack of education-specific benchmarks for pedagogical grammar by introducing a multi-tiered evaluation framework grounded in the Chinese Grammar Learning Manual. The approach formalizes pedagogical grammar recognition as a mapping problem and uses five task types—SINGLE, BATCH, SIM-GRA, CAT-GRA, and CON-INS—to assess grammar recognition, fine-grained distinctions, and resistance to interference. Key findings show larger-scale LLMs offer greater robustness to interference and complex data, but even them exhibit notable gaps in negative instance recognition and fine-grained discrimination, highlighting the need for better instructional alignment and benchmark rigor. The work provides actionable insights for educators and developers and establishes a foundation for future AI-assisted language instruction aligned with foreign language education goals, using synthetic yet pedagogy-grounded data and rigorous evaluation protocols.
Abstract
Purpose: The rapid emergence of large language models (LLMs) such as ChatGPT has significantly impacted foreign language education, yet their pedagogical grammar competence remains under-assessed. This paper introduces CPG-EVAL, the first dedicated benchmark specifically designed to evaluate LLMs' knowledge of pedagogical grammar within the context of foreign language instruction. Methodology: The benchmark comprises five tasks designed to assess grammar recognition, fine-grained grammatical distinction, categorical discrimination, and resistance to linguistic interference. Findings: Smaller-scale models can succeed in single language instance tasks, but struggle with multiple instance tasks and interference from confusing instances. Larger-scale models show better resistance to interference but still have significant room for accuracy improvement. The evaluation indicates the need for better instructional alignment and more rigorous benchmarks, to effectively guide the deployment of LLMs in educational contexts. Value: This study offers the first specialized, theory-driven, multi-tiered benchmark framework for systematically evaluating LLMs' pedagogical grammar competence in Chinese language teaching contexts. CPG-EVAL not only provides empirical insights for educators, policymakers, and model developers to better gauge AI's current abilities in educational settings, but also lays the groundwork for future research on improving model alignment, enhancing educational suitability, and ensuring informed decision-making concerning LLM integration in foreign language instruction.