Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CJEval: A Benchmark for Assessing Large Language Models Using Chinese Junior High School Exam Data

Qian-Wen Zhang, Haochen Wang, Fang Li, Siyu An, Lingfeng Qiao, Liangcai Gao, Di Yin, Xing Sun

TL;DR

CJEval, a benchmark based on Chinese Junior High School Exam Evaluations, is introduced, a benchmark based on Large Language Models' potential applications and a comprehensive analysis of their performance by fine-tuning on various educational tasks.

Abstract

Online education platforms have significantly transformed the dissemination of educational resources by providing a dynamic and digital infrastructure. With the further enhancement of this transformation, the advent of Large Language Models (LLMs) has elevated the intelligence levels of these platforms. However, current academic benchmarks provide limited guidance for real-world industry scenarios. This limitation arises because educational applications require more than mere test question responses. To bridge this gap, we introduce CJEval, a benchmark based on Chinese Junior High School Exam Evaluations. CJEval consists of 26,136 samples across four application-level educational tasks covering ten subjects. These samples include not only questions and answers but also detailed annotations such as question types, difficulty levels, knowledge concepts, and answer explanations. By utilizing this benchmark, we assessed LLMs' potential applications and conducted a comprehensive analysis of their performance by fine-tuning on various educational tasks. Extensive experiments and discussions have highlighted the opportunities and challenges of applying LLMs in the field of education.

CJEval: A Benchmark for Assessing Large Language Models Using Chinese Junior High School Exam Data

TL;DR

CJEval, a benchmark based on Chinese Junior High School Exam Evaluations, is introduced, a benchmark based on Large Language Models' potential applications and a comprehensive analysis of their performance by fine-tuning on various educational tasks.

Abstract

Online education platforms have significantly transformed the dissemination of educational resources by providing a dynamic and digital infrastructure. With the further enhancement of this transformation, the advent of Large Language Models (LLMs) has elevated the intelligence levels of these platforms. However, current academic benchmarks provide limited guidance for real-world industry scenarios. This limitation arises because educational applications require more than mere test question responses. To bridge this gap, we introduce CJEval, a benchmark based on Chinese Junior High School Exam Evaluations. CJEval consists of 26,136 samples across four application-level educational tasks covering ten subjects. These samples include not only questions and answers but also detailed annotations such as question types, difficulty levels, knowledge concepts, and answer explanations. By utilizing this benchmark, we assessed LLMs' potential applications and conducted a comprehensive analysis of their performance by fine-tuning on various educational tasks. Extensive experiments and discussions have highlighted the opportunities and challenges of applying LLMs in the field of education.
Paper Structure (22 sections, 3 equations, 5 figures, 3 tables)

This paper contains 22 sections, 3 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Online Examination System
  3. System Design
  4. Question-Based Tasks
  5. Knowledge Concept Tagging
  6. Question Difficulty Prediction
  7. Question Answering
  8. Question Generation
  9. CJEval Benchmark
  10. Experiments
  11. Accessed LLMs
  12. Experiment Setups
  13. Implementation Details
  14. Evaluation Metrics
  15. Results and Analysis
  16. ...and 7 more sections

Figures (5)

  • Figure 1: An overview of the design of online education systems reveals that next-generation systems are significantly more intelligent than classical ones.
  • Figure 2: Examples in CJEval. CJEval comprises authentic junior high school exam questions across 10 subjects, featuring various question types, question difficulty levels, knowledge concepts, and answer explanations. English translations are shown below the corresponding Chinese texts for better readability.
  • Figure 3: Detailed statistics on question difficulty and knowledge concepts.
  • Figure 4: The performance of models in KCT, QDP, and QG tasks across ten subjects.
  • Figure 5: The performance of models in different question types across ten subjects.