Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TongGu: Mastering Classical Chinese Understanding with Knowledge-Grounded Large Language Models

Jiahuan Cao, Dezhi Peng, Peirong Zhang, Yongxin Shi, Yang Liu, Kai Ding, Lianwen Jin

TL;DR

This work proposes TongGu, the first CCU-specific LLM, underpinned by three core contributions, aiming to unlock the full CCU potential of LLMs, and proposes Redundancy-Aware Tuning (RAT) to prevent catastrophic forgetting.

Abstract

Classical Chinese is a gateway to the rich heritage and wisdom of ancient China, yet its complexities pose formidable comprehension barriers for most modern people without specialized knowledge. While Large Language Models (LLMs) have shown remarkable capabilities in Natural Language Processing (NLP), they struggle with Classical Chinese Understanding (CCU), especially in data-demanding and knowledge-intensive tasks. In response to this dilemma, we propose \textbf{TongGu} (mean understanding ancient and modern), the first CCU-specific LLM, underpinned by three core contributions. First, we construct a two-stage instruction-tuning dataset ACCN-INS derived from rich classical Chinese corpora, aiming to unlock the full CCU potential of LLMs. Second, we propose Redundancy-Aware Tuning (RAT) to prevent catastrophic forgetting, enabling TongGu to acquire new capabilities while preserving its foundational knowledge. Third, we present a CCU Retrieval-Augmented Generation (CCU-RAG) technique to reduce hallucinations based on knowledge-grounding. Extensive experiments across 24 diverse CCU tasks validate TongGu's superior ability, underscoring the effectiveness of RAT and CCU-RAG. The model and dataset are available at \url{https://github.com/SCUT-DLVCLab/TongGu-LLM}.

TongGu: Mastering Classical Chinese Understanding with Knowledge-Grounded Large Language Models

TL;DR

This work proposes TongGu, the first CCU-specific LLM, underpinned by three core contributions, aiming to unlock the full CCU potential of LLMs, and proposes Redundancy-Aware Tuning (RAT) to prevent catastrophic forgetting.

Abstract

Classical Chinese is a gateway to the rich heritage and wisdom of ancient China, yet its complexities pose formidable comprehension barriers for most modern people without specialized knowledge. While Large Language Models (LLMs) have shown remarkable capabilities in Natural Language Processing (NLP), they struggle with Classical Chinese Understanding (CCU), especially in data-demanding and knowledge-intensive tasks. In response to this dilemma, we propose \textbf{TongGu} (mean understanding ancient and modern), the first CCU-specific LLM, underpinned by three core contributions. First, we construct a two-stage instruction-tuning dataset ACCN-INS derived from rich classical Chinese corpora, aiming to unlock the full CCU potential of LLMs. Second, we propose Redundancy-Aware Tuning (RAT) to prevent catastrophic forgetting, enabling TongGu to acquire new capabilities while preserving its foundational knowledge. Third, we present a CCU Retrieval-Augmented Generation (CCU-RAG) technique to reduce hallucinations based on knowledge-grounding. Extensive experiments across 24 diverse CCU tasks validate TongGu's superior ability, underscoring the effectiveness of RAT and CCU-RAG. The model and dataset are available at \url{https://github.com/SCUT-DLVCLab/TongGu-LLM}.
Paper Structure (22 sections, 1 equation, 10 figures, 13 tables, 1 algorithm)

This paper contains 22 sections, 1 equation, 10 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models
  4. Language Modeling for Classical Chinese Understanding
  5. Retrieval-Augmented Generation
  6. ACCN-INS Dataset
  7. TongGu
  8. Incremental Pre-training
  9. Two-stage Instruction Fine-tuning
  10. CCU-RAG
  11. Experiments
  12. Performance on C3bench
  13. Performance on Broader CCU Tasks
  14. Ablation Study
  15. Conclusion
  16. ...and 7 more sections

Figures (10)

  • Figure 1: Overview of the CCU instruction data generation pipeline from labeled and unlabeled text.
  • Figure 2: Data statistics of data-efficient tasks data in ACCN-INS dataset. (a) Distribution of sentence length. (b) Sample distribution for each task. Zoom in for a better view.
  • Figure 3: Overview of the training pipeline.
  • Figure 4: Two examples of reformatted knowledge-intensive tasks, with the difference being whether reference materials are provided.
  • Figure 5: Workflow of TongGu response with CCU-RAG.
  • ...and 5 more figures