Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DISCO: Diversifying Sample Condensation for Efficient Model Evaluation

Alexander Rubinstein, Benjamin Raible, Martin Gubri, Seong Joon Oh

TL;DR

DISCO addresses the escalating cost of evaluating large models by condensing evaluation data to a small, highly informative subset that maximizes inter-model disagreement. It combines a disagreement-based subset selection using $JSD$ or $PDS$ with a simple model-signature metamodel to map outputs on the condensed set to full benchmark performance, avoiding latent-parameter estimation. The approach delivers state-of-the-art efficiency on language and vision benchmarks, reducing data needs by roughly $99\%$ while maintaining competitive accuracy (e.g., $MAE\approx1.07\%p$ and rank $\approx0.987$ on MMLU; $MAE\approx0.63\%p$, rank $\approx0.969$ on ImageNet). By forgoing clustering and complex psychometric models, DISCO provides a scalable, practical tool for rapid model evaluation and tracking, with limitations mainly around distribution shifts and room for adaptive sampling in future work.

Abstract

Evaluating modern machine learning models has become prohibitively expensive. Benchmarks such as LMMs-Eval and HELM demand thousands of GPU hours per model. Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact. The typical approach follows two steps. First, select an anchor subset of data. Second, train a mapping from the accuracy on this subset to the final test result. The drawback is that anchor selection depends on clustering, which can be complex and sensitive to design choices. We argue that promoting diversity among samples is not essential; what matters is to select samples that $\textit{maximise diversity in model responses}$. Our method, $\textbf{Diversifying Sample Condensation (DISCO)}$, selects the top-k samples with the greatest model disagreements. This uses greedy, sample-wise statistics rather than global clustering. The approach is conceptually simpler. From a theoretical view, inter-model disagreement provides an information-theoretically optimal rule for such greedy selection. $\textbf{DISCO}$ shows empirical gains over prior methods, achieving state-of-the-art results in performance prediction across MMLU, Hellaswag, Winogrande, and ARC. Code is available here: https://github.com/arubique/disco-public.

DISCO: Diversifying Sample Condensation for Efficient Model Evaluation

TL;DR

DISCO addresses the escalating cost of evaluating large models by condensing evaluation data to a small, highly informative subset that maximizes inter-model disagreement. It combines a disagreement-based subset selection using or with a simple model-signature metamodel to map outputs on the condensed set to full benchmark performance, avoiding latent-parameter estimation. The approach delivers state-of-the-art efficiency on language and vision benchmarks, reducing data needs by roughly while maintaining competitive accuracy (e.g., and rank on MMLU; , rank on ImageNet). By forgoing clustering and complex psychometric models, DISCO provides a scalable, practical tool for rapid model evaluation and tracking, with limitations mainly around distribution shifts and room for adaptive sampling in future work.

Abstract

Evaluating modern machine learning models has become prohibitively expensive. Benchmarks such as LMMs-Eval and HELM demand thousands of GPU hours per model. Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact. The typical approach follows two steps. First, select an anchor subset of data. Second, train a mapping from the accuracy on this subset to the final test result. The drawback is that anchor selection depends on clustering, which can be complex and sensitive to design choices. We argue that promoting diversity among samples is not essential; what matters is to select samples that . Our method, , selects the top-k samples with the greatest model disagreements. This uses greedy, sample-wise statistics rather than global clustering. The approach is conceptually simpler. From a theoretical view, inter-model disagreement provides an information-theoretically optimal rule for such greedy selection. shows empirical gains over prior methods, achieving state-of-the-art results in performance prediction across MMLU, Hellaswag, Winogrande, and ARC. Code is available here: https://github.com/arubique/disco-public.

Paper Structure

This paper contains 28 sections, 7 theorems, 40 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Problem
  4. Solution
  5. Dataset selection
  6. Prior selection methods
  7. DISCO selection
  8. Performance prediction
  9. Prior prediction methods
  10. DISCO prediction
  11. Experiments
  12. Main results
  13. Factors analysis
  14. Results for vision domain
  15. Conclusion
  16. ...and 13 more sections

Key Result

Proposition 1

Let $\mathcal{D} = \{(x_i,y_i)\}_i^{N}$ be a test set and $m \sim \mathrm{Unif}\{1,\dots,M\}$ be the index of a uniformly chosen model. Let $f_{c}^m(x_i)\in[0,1]$ be the predictive probability for class $c$ of model $f^m$ on input $x_i$. We write $\widehat{y}_i^m$ for the categorical random variable where ${H}(\cdot)$ is entropy, $\mathrm{MI}(\cdot)$ is mutual information, and $\operatorname{JSD}(

Figures (7)

  • Figure 1: Imbalance. More evaluation budget is spent on less informative samples in test sets.
  • Figure 2: Problem overview. We aim at selecting a much smaller evaluation dataset than the original evaluation dataset, while keeping the estimated performances as close as possible. Figure \ref{['fig:dataset-condensation']} details the selection algorithm and the performance predictor.
  • Figure 3: DISCO overview. First, we select a subset of an evaluation dataset with the most informative samples. Second, we predict the performance of unseen models from their outputs on the selected samples.
  • Figure 4: True and estimated performance on MMLU. Scatter plot of performances of 40 models.
  • Figure 5: MMLU performance estimation vs. compression rates. Mean absolute error (MAE), measured in %p difference in accuracy, and the Spearman rank correlation between the true model ranking and the estimated model ranking are shown. At 100 samples, the results are identical to Table \ref{['tab:language-main']}. Main observations: DISCO hits a better efficiency-precision trade-off across all range of compression rates. For extreme compression rate, kNN is a better choice than random forest (RF).
  • ...and 2 more figures

Theorems & Definitions (17)

  • Proposition 1
  • Proposition 2
  • Definition 1: Jensen--Shannon divergence
  • Proposition 3
  • proof
  • Definition 2: Total variation
  • Proposition 4: JSD--TV sandwich bounds
  • proof
  • Remark 1
  • Definition 3: Envelope and spread, per class
  • ...and 7 more