Improving Model Evaluation using SMART Filtering of Benchmark Datasets

TL;DR

This paper tackles NLP evaluation reliability amid benchmark saturation and data leakage by introducing SMART filtering, a three-step, data-efficient approach to prune benchmark datasets of low-information content. By pre-filtering duplicates, removing easy and contaminated examples, and discarding similar items via embedding-based clustering, SMART filtering yields leaner datasets that preserve model rankings while improving alignment with human preferences. Empirical results on ARC, MMLU, and CommonsenseQA show substantial dataset reductions (up to 68.9%) with high Kendall's Tau stability and stronger correlations with ChatBot Arena Elo scores, indicating better reflection of real-world performance. The method offers practical benefits for revitalizing old benchmarks and guiding the creation of future, more challenging evaluation datasets, with clear pathways for iterative application and robustness checks.

Abstract

One of the most challenging problems facing NLP today is evaluation. Some of the most pressing issues pertain to benchmark saturation, data contamination, and diversity in the quality of test examples. To address these concerns, we propose Selection Methodology for Accurate, Reduced, and Targeted (SMART) filtering, a novel approach to select a high-quality subset of examples from existing benchmark datasets by systematically removing less informative and less challenging examples. Our approach applies three filtering criteria, removing (i) easy examples, (ii) data-contaminated examples, and (iii) examples that are similar to each other based on distance in an embedding space. We demonstrate the effectiveness of SMART on three multiple choice QA datasets, where our methodology increases efficiency by reducing dataset size by 48\% on average, while increasing Pearson correlation with rankings from ChatBot Arena, a more open-ended human evaluation setting. Our method enables us to be more efficient, whether using SMART to make new benchmarks more challenging or to revitalize older datasets, while still preserving the relative model rankings.

Figures (5)

• Figure 1: This figure illustrates our methodology. We remove easy, data contaminated, and/or similar examples from datasets to find a high-quality subset. The examples selected are from MMLU dataset.
• Figure 2: We remove question context and give answer-only prompt. If all models still predict the correct answer with high probability ($>0.8$), then we categorize that example as data contaminated.
• Figure 3: Cosine distances between SentenceBert embeddings for MMLU examples. The black vertical line is the threshold for identifying similar example pairs.
• Figure 4: Heatmap illustrating the degree of agreement between model predictions for ARC, MMLU and Commonsense QA datasets, as well as their SMART filtering variants. Notably, SMART filtering leads to a decrease in inter-model agreement, indicating that our approach helps in better differentiating the capabilities of the models.
• Figure 5: Distribution of cosine distances between SentenceBert embeddings for different values of k nearest neighbors for each example in the MMLU dataset. We find that threshold values obtained using different $k$ values for $k$-nearest neighbors converge after 100 neighbors.