Learning More from Less: Unlocking Internal Representations for Benchmark Compression

TL;DR

The paper tackles benchmark compression for large language models when historical evaluation data are scarce by proposing RepCore, which aligns heterogeneous hidden states into a unified latent space to recover the benchmark's geometry. It then selects a representative coreset via consensus clustering in this space and extrapolates full-benchmark performance for unseen models using a lightweight regressor trained on a single robust feature, achieving accurate estimates with as few as $|\,\mathcal{S}\|\le 10$ source models. Across five benchmarks and over 200 models, RepCore consistently improves ranking accuracy and estimation fidelity, and spectral analyses reveal separable, task-relevant structure beyond coarse difficulty signals. This work demonstrates that internal representations encode rich, interpretable structure that can be leveraged for scalable, robust benchmark compression with practical impact for rapid model evaluation and comparison.

Abstract

The prohibitive cost of evaluating Large Language Models (LLMs) necessitates efficient alternatives to full-scale benchmarking. Prevalent approaches address this by identifying a small coreset of items to approximate full-benchmark performance. However, existing methods must estimate a reliable item profile from response patterns across many source models, which becomes statistically unstable when the source pool is small. This dependency is particularly limiting for newly released benchmarks with minimal historical evaluation data. We argue that discrete correctness labels are a lossy view of the model's decision process and fail to capture information encoded in hidden states. To address this, we introduce REPCORE, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets. Using these subsets for performance extrapolation, REPCORE achieves precise estimation accuracy with as few as ten source models. Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy. Spectral analysis further indicates that the aligned representations contain separable components reflecting broad response tendencies and task-specific reasoning patterns.

Figures (3)

• Figure 1: Comparison of item representation paradigms in source-scarce regimes. Top: Output-based methods rely on sparse 0/1 signals that fail to preserve the geometric structure of the item space. Bottom: RepCore aligns heterogeneous hidden states into a unified latent space to recover fine-grained item structures for robust coreset selection.
• Figure 2: Overview of the RepCore framework. The pipeline proceeds in three phases: aligning heterogeneous hidden states into a unified latent space via model-specific projections and a shared MLP, selecting representative anchor items through consensus clustering, and extrapolating full-benchmark performance using a lightweight regressor.
• Figure 3: Factor-association analysis of the RepCore latent space on BBH. Dual axes quantify associations: the left axis reports Spearman's $\rho$ for continuous factors (blue), while the right axis denotes effect size $\epsilon^2$ for categorical factors (orange). Markers indicate principal components (Circle: PC$_1$, Square: PC$_2$, Triangle: PC$_3$). (a) Global Analysis: The primary axis (PC$_1$) is predominantly aligned with item difficulty ($\rho \approx 0.46$), while categorical task identities also show strong initial associations. (b) Stratified Analysis: After controlling for difficulty through binning, the embeddings maintain a high clustering effect by fine-grained Task Type ($\epsilon^2 \approx 0.54$), reinforcing the broader structural signal observed in coarse Ability Mode. This persistent structural signal confirms that the latent manifold encodes task-specific identities that transcend mere performance-based scalar signals.