Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WenyanGPT: A Large Language Model for Classical Chinese Tasks

Xinyu Yao, Mengdi Wang, Bo Chen, Xiaobing Zhao

TL;DR

Classical Chinese processing has lagged behind Modern Chinese NLP, lacking universal models and standardized benchmarks. WenyanGPT addresses this by continuing pre-training on a large Classical Chinese corpus with LLaMA3-8B-Chinese and applying domain-focused instruction fine-tuning, supported by WenyanBENCH for standardized evaluation. The authors release training data, instruction-finetuning resources, and the WenyanBench dataset, and show WenyanGPT achieves state-of-the-art performance across punctuation, POS tagging, NER, translation, and other tasks, outperforming existing open-source LLMs. This work provides a reproducible framework for domain-specific LLM development and offers practical tools to advance the preservation and study of Classical Chinese literature and culture.

Abstract

Classical Chinese, as the core carrier of Chinese culture, plays a crucial role in the inheritance and study of ancient literature. However, existing natural language processing models primarily optimize for Modern Chinese, resulting in inadequate performance on Classical Chinese. This paper presents a comprehensive solution for Classical Chinese language processing. By continuing pre-training and instruction fine-tuning on the LLaMA3-8B-Chinese model, we construct a large language model, WenyanGPT, which is specifically designed for Classical Chinese tasks. Additionally, we develop an evaluation benchmark dataset, WenyanBENCH. Experimental results on WenyanBENCH demonstrate that WenyanGPT significantly outperforms current advanced LLMs in various Classical Chinese tasks. We make the model's training data, instruction fine-tuning data\footnote, and evaluation benchmark dataset publicly available to promote further research and development in the field of Classical Chinese processing.

WenyanGPT: A Large Language Model for Classical Chinese Tasks

TL;DR

Classical Chinese processing has lagged behind Modern Chinese NLP, lacking universal models and standardized benchmarks. WenyanGPT addresses this by continuing pre-training on a large Classical Chinese corpus with LLaMA3-8B-Chinese and applying domain-focused instruction fine-tuning, supported by WenyanBENCH for standardized evaluation. The authors release training data, instruction-finetuning resources, and the WenyanBench dataset, and show WenyanGPT achieves state-of-the-art performance across punctuation, POS tagging, NER, translation, and other tasks, outperforming existing open-source LLMs. This work provides a reproducible framework for domain-specific LLM development and offers practical tools to advance the preservation and study of Classical Chinese literature and culture.

Abstract

Classical Chinese, as the core carrier of Chinese culture, plays a crucial role in the inheritance and study of ancient literature. However, existing natural language processing models primarily optimize for Modern Chinese, resulting in inadequate performance on Classical Chinese. This paper presents a comprehensive solution for Classical Chinese language processing. By continuing pre-training and instruction fine-tuning on the LLaMA3-8B-Chinese model, we construct a large language model, WenyanGPT, which is specifically designed for Classical Chinese tasks. Additionally, we develop an evaluation benchmark dataset, WenyanBENCH. Experimental results on WenyanBENCH demonstrate that WenyanGPT significantly outperforms current advanced LLMs in various Classical Chinese tasks. We make the model's training data, instruction fine-tuning data\footnote, and evaluation benchmark dataset publicly available to promote further research and development in the field of Classical Chinese processing.
Paper Structure (31 sections, 5 figures, 9 tables)

This paper contains 31 sections, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. PLMs
  4. Classical Chinese PLMs
  5. WenyanGPT
  6. Pre-training
  7. Supervised Fine-Tuning
  8. Data Selection and Initial Organization.
  9. Manual Design of Task Instructions and Model Expansion.
  10. Testing the Instruction Set and Optimizing Fine-Tuning Data.
  11. Generation and Supplementation of Specific Task Data.
  12. Data Integration and Output Validation.
  13. Benchmarking Classical Chinese Tasks
  14. WenyanBench.
  15. Tasks.
  16. ...and 16 more sections

Figures (5)

  • Figure 1: Examples of tasks from WenyanGPT. The model demonstrates advanced knowledge in Classical Chinese and shows strong performance in both Classical Chinese understanding and generation tasks.
  • Figure 2: Overall Training Framework of WenyanGPT.
  • Figure 3: Instruction Fine-Tuning Data Construction Process.
  • Figure 4: The F1 scores of WenYanGPT for the subcategories of understanding tasks (including Punctuation, Part-of-speech tagging, and Named Entity Recognition) on WenyanBench.
  • Figure 5: Radar plot showing model performance on WenyanBench, with values normalized to a 0-1 scale.