Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

C$^{3}$Bench: A Comprehensive Classical Chinese Understanding Benchmark for Large Language Models

Jiahuan Cao, Yongxin Shi, Dezhi Peng, Yang Liu, Lianwen Jin

TL;DR

C$^{3}$Bench introduces the first comprehensive multilingual CCU benchmark for LLMs, comprising $50{,}000$ text pairs across $5$ tasks from $10$ domains to assess classification, retrieval, NER, punctuation, and translation. The study evaluates $15$ representative models (open-source, closed-source, and supervised baselines) and presents a public leaderboard, revealing that current LLMs remain behind supervised methods and CCU tasks pose unique challenges requiring specialized knowledge. The benchmark construction combines domain-diverse data with rigorous annotation and quality control, enabling standardized cross-model comparisons and revealing imbalances in model capabilities across tasks. The work provides baseline results, actionable insights, and a clear direction for future CCU research and data-driven improvements in LLMs for classical Chinese understanding.

Abstract

Classical Chinese Understanding (CCU) holds significant value in preserving and exploration of the outstanding traditional Chinese culture. Recently, researchers have attempted to leverage the potential of Large Language Models (LLMs) for CCU by capitalizing on their remarkable comprehension and semantic capabilities. However, no comprehensive benchmark is available to assess the CCU capabilities of LLMs. To fill this gap, this paper introduces C$^{3}$bench, a Comprehensive Classical Chinese understanding benchmark, which comprises 50,000 text pairs for five primary CCU tasks, including classification, retrieval, named entity recognition, punctuation, and translation. Furthermore, the data in C$^{3}$bench originates from ten different domains, covering most of the categories in classical Chinese. Leveraging the proposed C$^{3}$bench, we extensively evaluate the quantitative performance of 15 representative LLMs on all five CCU tasks. Our results not only establish a public leaderboard of LLMs' CCU capabilities but also gain some findings. Specifically, existing LLMs are struggle with CCU tasks and still inferior to supervised models. Additionally, the results indicate that CCU is a task that requires special attention. We believe this study could provide a standard benchmark, comprehensive baselines, and valuable insights for the future advancement of LLM-based CCU research. The evaluation pipeline and dataset are available at \url{https://github.com/SCUT-DLVCLab/C3bench}.

C$^{3}$Bench: A Comprehensive Classical Chinese Understanding Benchmark for Large Language Models

TL;DR

CBench introduces the first comprehensive multilingual CCU benchmark for LLMs, comprising text pairs across tasks from domains to assess classification, retrieval, NER, punctuation, and translation. The study evaluates representative models (open-source, closed-source, and supervised baselines) and presents a public leaderboard, revealing that current LLMs remain behind supervised methods and CCU tasks pose unique challenges requiring specialized knowledge. The benchmark construction combines domain-diverse data with rigorous annotation and quality control, enabling standardized cross-model comparisons and revealing imbalances in model capabilities across tasks. The work provides baseline results, actionable insights, and a clear direction for future CCU research and data-driven improvements in LLMs for classical Chinese understanding.

Abstract

Classical Chinese Understanding (CCU) holds significant value in preserving and exploration of the outstanding traditional Chinese culture. Recently, researchers have attempted to leverage the potential of Large Language Models (LLMs) for CCU by capitalizing on their remarkable comprehension and semantic capabilities. However, no comprehensive benchmark is available to assess the CCU capabilities of LLMs. To fill this gap, this paper introduces Cbench, a Comprehensive Classical Chinese understanding benchmark, which comprises 50,000 text pairs for five primary CCU tasks, including classification, retrieval, named entity recognition, punctuation, and translation. Furthermore, the data in Cbench originates from ten different domains, covering most of the categories in classical Chinese. Leveraging the proposed Cbench, we extensively evaluate the quantitative performance of 15 representative LLMs on all five CCU tasks. Our results not only establish a public leaderboard of LLMs' CCU capabilities but also gain some findings. Specifically, existing LLMs are struggle with CCU tasks and still inferior to supervised models. Additionally, the results indicate that CCU is a task that requires special attention. We believe this study could provide a standard benchmark, comprehensive baselines, and valuable insights for the future advancement of LLM-based CCU research. The evaluation pipeline and dataset are available at \url{https://github.com/SCUT-DLVCLab/C3bench}.
Paper Structure (26 sections, 4 figures, 5 tables)

This paper contains 26 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Large language models
  4. Automatically classical Chinese understanding
  5. Chinese NLP evaluation benchmarks
  6. Classical Chinese benchmarks
  7. C$^{3}$Bench
  8. Task definition
  9. Benchmark construction
  10. Data statistics
  11. Experiments
  12. Models
  13. Open-source models:
  14. Closed-source models:
  15. Supervised methods:
  16. ...and 11 more sections

Figures (4)

  • Figure 1: The performance of several representative evaluated models on C$^{3}$Bench.
  • Figure 2: Domain distribution of the dataset.
  • Figure 3: Distribution of sentence length.
  • Figure 4: Overall ranking of Classical Chinese Understanding capability for all the evaluated models.