Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddings
Andrew M. Bean, Nabeel Seedat, Shengzhuang Chen, Jonathan Richard Schwarz
TL;DR
The paper tackles the cost and cold-start limitations of evaluating large language models on full benchmarks by proposing an item-centric benchmark subset selection paradigm. It introduces Scales++, which encodes each benchmark item as a 16-dimensional cognitive-demands embedding, reduces dimensionality with UMAP, clusters to select a diverse subset, and predicts full benchmark scores via a hybrid estimator; annotation costs are further reduced with Scales++ Lite, a GNN-based predictor trained on GPT-4o-derived labels. The main contributions are the item-centric paradigm, the cognitive-scales embedding framework, and the amortized annotation strategy that enables efficient benchmarking with minimal data (e.g., 0.5% of items achieving MAE around $2.9\%$ on the Open LLM Leaderboard). The work demonstrates substantial upfront-cost reductions (over $18x$) without meaningful fidelity loss, enables better cold-start evaluation, and offers interpretable benchmarking by examining cognitive-demand profiles across tasks.
Abstract
The prohibitive cost of evaluating large language models (LLMs) on comprehensive benchmarks necessitates the creation of small yet representative data subsets (i.e., tiny benchmarks) that enable efficient assessment while retaining predictive fidelity. Current methods for this task operate under a model-centric paradigm, selecting benchmarking items based on the collective performance of existing models. Such approaches are limited by large upfront costs, an inability to immediately handle new benchmarks (`cold-start'), and the fragile assumption that future models will share the failure patterns of their predecessors. In this work, we challenge this paradigm and propose a item-centric approach to benchmark subset selection, arguing that selection should be based on the intrinsic properties of the task items themselves, rather than on model-specific failure patterns. We instantiate this item-centric efficient benchmarking approach via a novel method, Scales++, where data selection is based on the cognitive demands of the benchmark samples. Empirically, we show Scales++ reduces the upfront selection cost by over 18x while achieving competitive predictive fidelity. On the Open LLM Leaderboard, using just a 0.5\% data subset, we predict full benchmark scores with a 2.9% mean absolute error. We demonstrate that this item-centric approach enables more efficient model evaluation without significant fidelity degradation, while also providing better cold-start performance and more interpretable benchmarking.