Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ANGO: A Next-Level Evaluation Benchmark For Generation-Oriented Language Models In Chinese Domain

Bingchao Wang

TL;DR

ANGO tackles reliability and interpretability gaps in generation-oriented language-model evaluation by introducing a Chinese benchmark defined by a Keypoint-based annotation and a nine-level difficulty scale derived from real human performance. It combines exclusive sampling and a dynamic evaluation framework to minimize data leakage and enable rapid testset iteration, providing richer diagnostics than prior benchmarks. Empirical results show ANGO poses a stronger challenge and yields nuanced insights across keypoints, difficulty levels, and human-value metrics, with notable implications for Chinese-domain language models. Overall, ANGO offers a practical, adaptable platform for advancing evaluation and development of generation-oriented language models in Chinese contexts.

Abstract

Recently, various Large Language Models (LLMs) evaluation datasets have emerged, but most of them have issues with distorted rankings and difficulty in model capabilities analysis. Addressing these concerns, this paper introduces ANGO, a Chinese multi-choice question evaluation benchmark. ANGO proposes Keypoint categorization standard for the first time, each question in ANGO can correspond to multiple keypoints, effectively enhancing interpretability of evaluation results. Base on performance of real humans, we build a quantifiable question difficulty standard and divide ANGO questions into 9 difficulty levels, which provide more precise guidance for model training. To minimize data leakage impact and fully leverage ANGO's innovative features, we have engineered exclusive sampling strategies and a new evaluation framework that support swift testset iteration. Our experiments demonstrate that ANGO poses a stronger challenge to models and reveals more details in evaluation result compared to existing benchmarks.

ANGO: A Next-Level Evaluation Benchmark For Generation-Oriented Language Models In Chinese Domain

TL;DR

ANGO tackles reliability and interpretability gaps in generation-oriented language-model evaluation by introducing a Chinese benchmark defined by a Keypoint-based annotation and a nine-level difficulty scale derived from real human performance. It combines exclusive sampling and a dynamic evaluation framework to minimize data leakage and enable rapid testset iteration, providing richer diagnostics than prior benchmarks. Empirical results show ANGO poses a stronger challenge and yields nuanced insights across keypoints, difficulty levels, and human-value metrics, with notable implications for Chinese-domain language models. Overall, ANGO offers a practical, adaptable platform for advancing evaluation and development of generation-oriented language models in Chinese contexts.

Abstract

Recently, various Large Language Models (LLMs) evaluation datasets have emerged, but most of them have issues with distorted rankings and difficulty in model capabilities analysis. Addressing these concerns, this paper introduces ANGO, a Chinese multi-choice question evaluation benchmark. ANGO proposes Keypoint categorization standard for the first time, each question in ANGO can correspond to multiple keypoints, effectively enhancing interpretability of evaluation results. Base on performance of real humans, we build a quantifiable question difficulty standard and divide ANGO questions into 9 difficulty levels, which provide more precise guidance for model training. To minimize data leakage impact and fully leverage ANGO's innovative features, we have engineered exclusive sampling strategies and a new evaluation framework that support swift testset iteration. Our experiments demonstrate that ANGO poses a stronger challenge to models and reveals more details in evaluation result compared to existing benchmarks.
Paper Structure (38 sections, 3 equations, 8 figures, 3 tables)

This paper contains 38 sections, 3 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Challenges:
  3. Contributions:
  4. Data
  5. Data Collection
  6. Data Source
  7. Data Preprocessing
  8. Data Distribution
  9. Keypoint Tree
  10. Difficulty Measurement
  11. Sample Strategy
  12. Few-Shot History Sample
  13. Testset Sample
  14. Build $K_s$ Vectors:
  15. Initialize Testset:
  16. ...and 23 more sections

Figures (8)

  • Figure 1: ANGO: A Next-level evaluation benchmark for Generation-Oriented language models
  • Figure 2: The Data Preprocess Flow: From 674,675 source data to 71,149 clean data
  • Figure 3: Keypoint Tree: The portion of the area represents the data size, including 171 keypoints and 4 levels
  • Figure 4: Difficulty Distribution: x-axis stands for difficulty level and y-axis means the question count
  • Figure 5: Prompt Example: English translations are shown below the corresponding Chinese text for better readability.
  • ...and 3 more figures