Can the capability of Large Language Models be described by human ability? A Meta Study
Mingrui Zan, Yunquan Zhang, Boyang Zhang, Fangming Liu, Daning Cheng
TL;DR
This work investigates whether LLM capabilities can be described by human ability classifications. By compiling performance data from over 80 models across 37 benchmarks and clustering benchmark rankings using Spearman's rank correlation, the authors compare machine capabilities with a human-ability taxonomy. They find that some sub-10B capabilities align with human abilities, but many human-side correlations do not transfer to LLMs, and certain capabilities emerge or disappear as model scale changes. The study offers practical guidance for benchmark design, evaluation, and task allocation of LLMs in real-world settings.
Abstract
Users of Large Language Models (LLMs) often perceive these models as intelligent entities with human-like capabilities. However, the extent to which LLMs' capabilities truly approximate human abilities remains a topic of debate. In this paper, to characterize the capabilities of LLMs in relation to human capabilities, we collected performance data from over 80 models across 37 evaluation benchmarks. The evaluation benchmarks are categorized into 6 primary abilities and 11 sub-abilities in human aspect. Then, we then clustered the performance rankings into several categories and compared these clustering results with classifications based on human ability aspects. Our findings lead to the following conclusions: 1. We have confirmed that certain capabilities of LLMs with fewer than 10 billion parameters can indeed be described using human ability metrics; 2. While some abilities are considered interrelated in humans, they appear nearly uncorrelated in LLMs; 3. The capabilities possessed by LLMs vary significantly with the parameter scale of the model.