Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AC-EVAL: Evaluating Ancient Chinese Language Understanding in Large Language Models

Yuting Wei, Yuanxing Xu, Xinru Wei, Simin Yang, Yangfu Zhu, Yuqing Li, Di Liu, Bin Wu

TL;DR

An innovative benchmark designed to assess the advanced knowledge and reasoning capabilities of LLMs within the context of ancient Chinese, AC-EVAL is structured across three levels of difficulty reflecting different facets of language comprehension.

Abstract

Given the importance of ancient Chinese in capturing the essence of rich historical and cultural heritage, the rapid advancements in Large Language Models (LLMs) necessitate benchmarks that can effectively evaluate their understanding of ancient contexts. To meet this need, we present AC-EVAL, an innovative benchmark designed to assess the advanced knowledge and reasoning capabilities of LLMs within the context of ancient Chinese. AC-EVAL is structured across three levels of difficulty reflecting different facets of language comprehension: general historical knowledge, short text understanding, and long text comprehension. The benchmark comprises 13 tasks, spanning historical facts, geography, social customs, art, philosophy, classical poetry and prose, providing a comprehensive assessment framework. Our extensive evaluation of top-performing LLMs, tailored for both English and Chinese, reveals a substantial potential for enhancing ancient text comprehension. By highlighting the strengths and weaknesses of LLMs, AC-EVAL aims to promote their development and application forward in the realms of ancient Chinese language education and scholarly research. The AC-EVAL data and evaluation code are available at https://github.com/yuting-wei/AC-EVAL.

AC-EVAL: Evaluating Ancient Chinese Language Understanding in Large Language Models

TL;DR

An innovative benchmark designed to assess the advanced knowledge and reasoning capabilities of LLMs within the context of ancient Chinese, AC-EVAL is structured across three levels of difficulty reflecting different facets of language comprehension.

Abstract

Given the importance of ancient Chinese in capturing the essence of rich historical and cultural heritage, the rapid advancements in Large Language Models (LLMs) necessitate benchmarks that can effectively evaluate their understanding of ancient contexts. To meet this need, we present AC-EVAL, an innovative benchmark designed to assess the advanced knowledge and reasoning capabilities of LLMs within the context of ancient Chinese. AC-EVAL is structured across three levels of difficulty reflecting different facets of language comprehension: general historical knowledge, short text understanding, and long text comprehension. The benchmark comprises 13 tasks, spanning historical facts, geography, social customs, art, philosophy, classical poetry and prose, providing a comprehensive assessment framework. Our extensive evaluation of top-performing LLMs, tailored for both English and Chinese, reveals a substantial potential for enhancing ancient text comprehension. By highlighting the strengths and weaknesses of LLMs, AC-EVAL aims to promote their development and application forward in the realms of ancient Chinese language education and scholarly research. The AC-EVAL data and evaluation code are available at https://github.com/yuting-wei/AC-EVAL.
Paper Structure (22 sections, 6 figures, 7 tables)

This paper contains 22 sections, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Chinese benchmarks for LLMs
  4. Ancient Chinese benchmarks
  5. AC-EVAL Overview
  6. Design Principles
  7. Data Collection
  8. Evaluation
  9. Experiment
  10. Setup
  11. Models
  12. Results
  13. Zero-shot AO
  14. Few-shot AO
  15. Chain-of-thought
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Overview of AC-EVAL.
  • Figure 2: Illustrative few-shot AO prompts from AC-EVAL with corresponding English translations for better readability.
  • Figure 3: Average accuracy for each category in various settings with different model sizes. Here, we have omitted the category names and shot types for brevity.
  • Figure 4: Illustration of the Annotation Process: An Example of Social Customs Data Annotation
  • Figure 5: Illustrative zero-shot COT prompts from AC-EVAL with corresponding English translations for better readability.
  • ...and 1 more figures