HSKBenchmark: Modeling and Benchmarking Chinese Second Language Acquisition in Large Language Models through Curriculum Tuning

TL;DR

This work introduces HSKBenchmark, the first benchmark for staged modeling and writing assessment of Chinese SLA in LLMs, using level-based textbooks (HSK3-6), 16K synthetic instruction data for grammar items, and a curriculum-tuning framework to simulate progressive SLA trajectories. It couples a linguistically-grounded evaluation system with HSKAgent, an automated evaluator trained on a large learner corpus, enabling dynamic, level-aware assessment of writing quality. Experimental results show curriculum-tuned LLMs achieve near-human writing performance and exhibit human-like acquisition patterns, while preserving general Chinese abilities and L1 proficiency. The framework, data, and tools provide a foundation for further research on language acquisition modeling and LLM interpretability, with public release of resources for broader adoption.

Abstract

Language acquisition is vital to revealing the nature of human language intelligence and has recently emerged as a promising perspective for improving the interpretability of large language models (LLMs). However, it is ethically and practically infeasible to conduct experiments that require controlling human learners' language inputs. This poses challenges for the verifiability and scalability of language acquisition modeling, particularly in Chinese second language acquisition (SLA). While LLMs provide a controllable and reproducible alternative, a systematic benchmark to support phase-wise modeling and assessment is still lacking. In this paper, we present HSKBenchmark, the first benchmark for staged modeling and writing assessment of LLMs in Chinese SLA. It covers HSK levels 3 to 6 and includes authentic textbooks with 6.76 million tokens, 16K synthetic instruction samples, 30 test topics, and a linguistically grounded evaluation system. To simulate human learning trajectories, we introduce a curriculum-tuning framework that trains models from beginner to advanced levels. An evaluation system is created to examine level-based grammar coverage, writing errors, lexical and syntactic complexity, and holistic scoring. We also build HSKAgent, fine-tuned on 10K learner compositions. Extensive experimental results demonstrate that HSKBenchmark not only models Chinese SLA effectively, but also serves as a reliable benchmark for dynamic writing assessment in LLMs. Our fine-tuned LLMs have writing performance on par with advanced human learners and exhibit human-like acquisition characteristics. The HSKBenchmark, HSKAgent, and checkpoints serve as foundational tools and resources, with the potential to pave the way for future research on language acquisition modeling and LLMs interpretability. Code and data are publicly available at: https://github.com/CharlesYang030/HSKB.

Figures (9)

• Figure 1: Illustration of Chinese SLA modeling and dynamic writing assessment in LLMs.
• Figure 2: Illustration of our HSKBenchmark. It contains the level-based training data, the curriculum-tuning framework, the linguistically-grounded evaluation system and the HSKAgent.
• Figure 3: Comparison between our LLMs and those trained on the shuffled dataset in overall average scores. CT refers to the curriculum tuning.
• Figure 4: The performance of Llama2 on MMLU and C-Eval.
• Figure 5: The prompt of generating instruction data based on the level-based grammar items (Appendix C).