CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models

TL;DR

CLongEval addresses the gap in evaluating long-context Chinese LLMs by introducing a 7-task benchmark with 7,267 examples covering full- and partial-context information acquisition and reasoning. It spans small to large context windows (1K–100K) and includes diverse data sources, including human-annotated and GPT-annotated tasks, to mirror real-world usage. The paper benchmarks eight models (four commercial, four open-source) and reveals consistent advantages for commercial systems in extraction-heavy tasks, while highlighting pronounced weaknesses in open-source models under long-context conditions, especially for stacked tasks and retrieval-based challenges. It also analyzes the influence of answer position and task-specific degradation, offering insights into where long-context capabilities remain challenging and where future improvements should focus.

Abstract

Developing Large Language Models (LLMs) with robust long-context capabilities has been the recent research focus, resulting in the emergence of long-context LLMs proficient in Chinese. However, the evaluation of these models remains underdeveloped due to a lack of benchmarks. To address this gap, we present CLongEval, a comprehensive Chinese benchmark for evaluating long-context LLMs. CLongEval is characterized by three key features: (1) Sufficient data volume, comprising 7 distinct tasks and 7,267 examples; (2) Broad applicability, accommodating to models with context windows size from 1K to 100K; (3) High quality, with over 2,000 manually annotated question-answer pairs in addition to the automatically constructed labels. With CLongEval, we undertake a comprehensive assessment of 6 open-source long-context LLMs and 2 leading commercial counterparts that feature both long-context abilities and proficiency in Chinese. We also provide in-depth analysis based on the empirical results, trying to shed light on the critical capabilities that present challenges in long-context settings. The dataset, evaluation scripts, and model outputs are released.

Figures (4)

• Figure 1: The evaluation framework of CLongEval. The seven test tasks in CLongEval are designed to comprehensively assess two important capabilities of long-context LLMs: information acquisition and reasoning.
• Figure 2: Effects of referenced chunk positions. The analyzed samples range from 10K to 60K in context length.
• Figure 3: Performance change analysis of GPT-4-Turbo on StNLab.
• Figure 4: Performance change analysis of Moonshot-v1 on StNLab.