Less is more: Not all samples are effective for evaluation

TL;DR

The paper addresses the cost and reliability challenges of evaluating domain-specific LLMs with large benchmarks by introducing a history-free compression framework. It jointly learns domain semantics via full-parameter fine-tuning, builds task-aware semantic embeddings, and uses adaptive sampling guided by a Dataset X-ray analysis to remove redundancy. On a large 3GPP QA benchmark, the approach compresses the test set to a small fraction without sacrificing model rankings, achieving a high Spearman correlation with the full benchmark and minimal accuracy fluctuation. This enables rapid, low-cost evaluation for cold-start domains and provides seed data for contamination detection, with broad practical impact for benchmarking vertical-domain LLMs.

Abstract

The versatility of Large Language Models (LLMs) in vertical domains has spurred the development of numerous specialized evaluation benchmarks. However, these benchmarks often suffer from significant semantic redundancy and impose high computational costs during evaluation. Existing compression methods, such as tinyBenchmarks depend critically on correctness labels from multiple historical models evaluated on the full test set, making them inapplicable in cold-start scenarios, such as the introduction of a new task, domain, or model with no prior evaluation history. To address this limitation, we propose a history-free test set compression framework that requires no prior model performance data. Our method begins by fine-tuning a base LLM on a small amount of domain-specific data to internalize task-relevant semantics. It then generates high-level semantic embeddings for all original test samples using only their raw textual content. In this domain-adapted embedding space, we perform task-aware clustering and introduce a novel dataset X-ray mechanism that analyzes cluster geometry to dynamically calibrate the compression intensity based on the intrinsic redundancy of the benchmark. Experiments on professional-domain dataset, notably a large-scale 3GPP communications benchmark, demonstrate that our approach effectively identifies and removes redundant samples, reducing evaluation cost by over 90% while preserving high fidelity to the full benchmark.

Figures (2)

• Figure 1: The pipeline of our proposed method.
• Figure 2: Clustering of 3GPP testset by fine-tuned Qwen3-1.7B.