Through the Lens of Core Competency: Survey on Evaluation of Large Language Models

TL;DR

The paper tackles the fragmented landscape of LLM evaluation by proposing a core competency framework that unifies benchmarks across knowledge, reasoning, reliability, and safety. It systematically categorizes hundreds of tasks into structured competencies and sub-areas, highlighting representative datasets and metrics for each. The contributions include a taxonomy for organizing evaluation, an extensible project linking tasks to competencies, and a roadmap for future directions in sentiment, planning, and coding. This framework aims to streamline evaluation, support comparability, and guide future research as LLM capabilities continue to evolve.

Abstract

From pre-trained language model (PLM) to large language model (LLM), the field of natural language processing (NLP) has witnessed steep performance gains and wide practical uses. The evaluation of a research field guides its direction of improvement. However, LLMs are extremely hard to thoroughly evaluate for two reasons. First of all, traditional NLP tasks become inadequate due to the excellent performance of LLM. Secondly, existing evaluation tasks are difficult to keep up with the wide range of applications in real-world scenarios. To tackle these problems, existing works proposed various benchmarks to better evaluate LLMs. To clarify the numerous evaluation tasks in both academia and industry, we investigate multiple papers concerning LLM evaluations. We summarize 4 core competencies of LLM, including reasoning, knowledge, reliability, and safety. For every competency, we introduce its definition, corresponding benchmarks, and metrics. Under this competency architecture, similar tasks are combined to reflect corresponding ability, while new tasks can also be easily added into the system. Finally, we give our suggestions on the future direction of LLM's evaluation.