Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone

Nishant Subramani, Alfredo Gomez, Mona Diab

TL;DR

SimBA introduces a three-phase framework (stalk, prowl, pounce) to simplify benchmark analysis of language models using only performance matrices. By normalizing to percent-above-random and analyzing inter-dataset and inter-model relationships, SimBA discovers small representative subsets that cover benchmark behavior by >95% and preserve model rankings. It then predicts performance on held-out models with near-zero error using simple regressors, demonstrating efficiency gains for dataset curation and model evaluation across HELM, MMLU, and BigBenchLite. The approach offers a lightweight, robust tool for practitioners and dataset creators to validate benchmark structure while reducing evaluation costs, with open-source code available at the project URL.

Abstract

Modern language models are evaluated on large benchmarks, which are difficult to make sense of, especially for model selection. Looking at the raw evaluation numbers themselves using a model-centric lens, we propose SimBA, a three phase framework to Simplify Benchmark Analysis. The three phases of SimBA are: stalk, where we conduct dataset & model comparisons, prowl, where we discover a representative subset, and pounce, where we use the representative subset to predict performance on a held-out set of models. Applying SimBA to three popular LM benchmarks: HELM, MMLU, and BigBenchLite reveals that across all three benchmarks, datasets and models relate strongly to one another (stalk). We develop an representative set discovery algorithm which covers a benchmark using raw evaluation scores alone. Using our algorithm, we find that with 6.25% (1/16), 1.7% (1/58), and 28.4% (21/74) of the datasets for HELM, MMLU, and BigBenchLite respectively, we achieve coverage levels of at least 95% (prowl). Additionally, using just these representative subsets, we can both preserve model ranks and predict performance on a held-out set of models with near zero mean-squared error (pounce). Taken together, SimBA can help model developers improve efficiency during model training and dataset creators validate whether their newly created dataset differs from existing datasets in a benchmark. Our code is open source, available at https://github.com/nishantsubramani/simba.

SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alone

TL;DR

SimBA introduces a three-phase framework (stalk, prowl, pounce) to simplify benchmark analysis of language models using only performance matrices. By normalizing to percent-above-random and analyzing inter-dataset and inter-model relationships, SimBA discovers small representative subsets that cover benchmark behavior by >95% and preserve model rankings. It then predicts performance on held-out models with near-zero error using simple regressors, demonstrating efficiency gains for dataset curation and model evaluation across HELM, MMLU, and BigBenchLite. The approach offers a lightweight, robust tool for practitioners and dataset creators to validate benchmark structure while reducing evaluation costs, with open-source code available at the project URL.

Abstract

Modern language models are evaluated on large benchmarks, which are difficult to make sense of, especially for model selection. Looking at the raw evaluation numbers themselves using a model-centric lens, we propose SimBA, a three phase framework to Simplify Benchmark Analysis. The three phases of SimBA are: stalk, where we conduct dataset & model comparisons, prowl, where we discover a representative subset, and pounce, where we use the representative subset to predict performance on a held-out set of models. Applying SimBA to three popular LM benchmarks: HELM, MMLU, and BigBenchLite reveals that across all three benchmarks, datasets and models relate strongly to one another (stalk). We develop an representative set discovery algorithm which covers a benchmark using raw evaluation scores alone. Using our algorithm, we find that with 6.25% (1/16), 1.7% (1/58), and 28.4% (21/74) of the datasets for HELM, MMLU, and BigBenchLite respectively, we achieve coverage levels of at least 95% (prowl). Additionally, using just these representative subsets, we can both preserve model ranks and predict performance on a held-out set of models with near zero mean-squared error (pounce). Taken together, SimBA can help model developers improve efficiency during model training and dataset creators validate whether their newly created dataset differs from existing datasets in a benchmark. Our code is open source, available at https://github.com/nishantsubramani/simba.
Paper Structure (42 sections, 15 equations, 3 figures, 6 tables, 1 algorithm)

This paper contains 42 sections, 15 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Stalk: Dataset & Model Comparison
  3. Dataset Comparison
  4. Model Comparison
  5. Prowl: Representative Dataset Discovery
  6. Dataset Similarity
  7. Discovering a Representative Subset
  8. Baselines:
  9. Representative Subset Evaluation
  10. Pounce: Performance Prediction
  11. Regularized Linear Regression
  12. KNN Regression
  13. MLP Regression
  14. Performance Prediction Evaluation
  15. Experiments
  16. ...and 27 more sections

Figures (3)

  • Figure 1: An overview of SimBA, our three phase analysis framework. Its three phases are: stalk: dataset & model comparison, prowl: representative dataset discovery, and pounce: performance prediction.
  • Figure 2: Here, we measure the coverage ($\eta$) across HELM, MMLU, and BigBenchLite as our representative subset $S$ grows. We report performance for all three baselines and all nine similarity measures discussed in \ref{['sec: identifying_the_representative_dataset']}.
  • Figure 3: We measure the mean squared error (MSE) on a held-out test set of models of regressors trained on a representative subset $S$ using Minkowski (p=3) as sim. This is measured for HELM, MMLU, and BigBenchLite for the three regressors we experiment with in \ref{['sec: performance_prediction']}. Additionally, we repeat this experiment after adding two magnitudes of noise to the training set of $B$ according to \ref{['sec: robustness']} ($\sigma = 0.05$ and $\sigma = 0.1$). Lower scores are better.