Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks

Yixin Cao, Shibo Hong, Xinze Li, Jiahao Ying, Yubo Ma, Haiyuan Liang, Yantao Liu, Zijun Yao, Xiaozhi Wang, Dan Huang, Wenxuan Zhang, Lifu Huang, Muhao Chen, Lei Hou, Qianru Sun, Xingjun Ma, Zuxuan Wu, Min-Yen Kan, David Lo, Qi Zhang, Heng Ji, Jing Jiang, Juanzi Li, Aixin Sun, Xuanjing Huang, Tat-Seng Chua, Yu-Gang Jiang

TL;DR

The paper analyzes evaluation generalization in the LLM era, arguing that bounded benchmarks cannot keep pace with unbounded model capabilities. It advocates two major transitions—task-specific to capability-based evaluation and manual to automated evaluation—and presents a framework for living benchmarks, including dynamic data, LLM-based evaluators, and auto-curation pipelines. It discusses open challenges across capability-based, automated, and generalizable evaluation, and outlines directions such as predictive evaluation, adaptive datasets, and see-the-whole-from-a-part evaluators. By detailing integrated benchmarks and agent-centric evaluation while proposing community-driven maintenance, the work aims to guide robust, scalable, and fair assessment of next-generation LLMs.

Abstract

Large Language Models (LLMs) are advancing at an amazing speed and have become indispensable across academia, industry, and daily applications. To keep pace with the status quo, this survey probes the core challenges that the rise of LLMs poses for evaluation. We identify and analyze two pivotal transitions: (i) from task-specific to capability-based evaluation, which reorganizes benchmarks around core competencies such as knowledge, reasoning, instruction following, multi-modal understanding, and safety; and (ii) from manual to automated evaluation, encompassing dynamic dataset curation and "LLM-as-a-judge" scoring. Yet, even with these transitions, a crucial obstacle persists: the evaluation generalization issue. Bounded test sets cannot scale alongside models whose abilities grow seemingly without limit. We will dissect this issue, along with the core challenges of the above two transitions, from the perspectives of methods, datasets, evaluators, and metrics. Due to the fast evolving of this field, we will maintain a living GitHub repository (links are in each section) to crowd-source updates and corrections, and warmly invite contributors and collaborators.

Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks

TL;DR

The paper analyzes evaluation generalization in the LLM era, arguing that bounded benchmarks cannot keep pace with unbounded model capabilities. It advocates two major transitions—task-specific to capability-based evaluation and manual to automated evaluation—and presents a framework for living benchmarks, including dynamic data, LLM-based evaluators, and auto-curation pipelines. It discusses open challenges across capability-based, automated, and generalizable evaluation, and outlines directions such as predictive evaluation, adaptive datasets, and see-the-whole-from-a-part evaluators. By detailing integrated benchmarks and agent-centric evaluation while proposing community-driven maintenance, the work aims to guide robust, scalable, and fair assessment of next-generation LLMs.

Abstract

Large Language Models (LLMs) are advancing at an amazing speed and have become indispensable across academia, industry, and daily applications. To keep pace with the status quo, this survey probes the core challenges that the rise of LLMs poses for evaluation. We identify and analyze two pivotal transitions: (i) from task-specific to capability-based evaluation, which reorganizes benchmarks around core competencies such as knowledge, reasoning, instruction following, multi-modal understanding, and safety; and (ii) from manual to automated evaluation, encompassing dynamic dataset curation and "LLM-as-a-judge" scoring. Yet, even with these transitions, a crucial obstacle persists: the evaluation generalization issue. Bounded test sets cannot scale alongside models whose abilities grow seemingly without limit. We will dissect this issue, along with the core challenges of the above two transitions, from the perspectives of methods, datasets, evaluators, and metrics. Due to the fast evolving of this field, we will maintain a living GitHub repository (links are in each section) to crowd-source updates and corrections, and warmly invite contributors and collaborators.

Paper Structure

This paper contains 41 sections, 1 equation, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Core Capabilities and Datasets
  3. Knowledge Evaluation
  4. Reasoning Evaluation
  5. Mathematics Evaluation
  6. Coding Evaluation
  7. Logic Reasoning
  8. Commonsense Reasoning
  9. Long Context Reasoning
  10. Planning
  11. Miscellaneous Reasoning
  12. Instruction Following
  13. Multi-modal Evaluation
  14. Visual Question Answering
  15. Visual Document Comprehension
  16. ...and 26 more sections

Figures (6)

  • Figure 1: Illustration of two transitions in the field of evaluation for LLMs.
  • Figure 2: The logic of reviewing knowledge evaluation.
  • Figure 3: Mathematics benchmarks in chronological order. The y-axis represents four difficulty levels: Primary School, High School, University, and Olympiad. The area represents the size of each dataset.
  • Figure 4: Coding benchmarks, the programming languages (left), and coding tasks (right) that each benchmark use. We assign one color to each benchmark.
  • Figure 5: Illustration of instruction following's paradigm shift: from tasks, where tasks are gathered and their descriptions are used as instructions, to user needs where user prompts are regarded as "fine-grained" tasks' instructions.
  • ...and 1 more figures